Reading mode: Light Dark

Firmware idea: Distribute images across flash cards

Go to last post
H

Horshack

Forum Pro
Messages
11,231
Solutions
28
Reaction score
12,594
Location
US
This has probably been thought of before but just in case...

Pro bodies and now more recently prosumer bodies (D7000) have dual card slots. Camera firmware provides different options on how the two cards are utilized:
  • Redundancy (write images to both cards)
  • Dedicated purpose (raw on one, jpeg on other)
  • Capacity aggregation (when one card fills up, camera siwtches/overflows to the second card).
How about a parallel distribution option, where every image alternates to the other card, producing a 2x increase in buffer drain throughput and thus an effective increase in buffer capacity? Originally I had thought RAID 0 to stripe each image across the two cards but since fast buffer drain is only needed for continuous shooting it's simpler to just distribute entire images across the cards. Plus there's inherent redundancy in alternating on an entire-image basis as it would likely be used for continuous shooting and so even if a card fails you'll have every other image on the non-failing card from the same shooting sequence.

Performance scaling would be 2x when the cards of equal speeds, even if one is CF and the other SD. If there is a speed differential between cards then firmware could alter the image ratio accordingly to balance the throughput. This could be done on a dynamic basis to accommodate asymmetry in card performance resulting from:
  • Base throughput differences between the cards
  • Different image sizes (worst-case distribution of image sizes in a sequence)
  • Outlier latencies on an individual card due to internal flash operations
Regarding hardware support for this, I don't think there's an internal bandwidth or DMA serialization issue since it's clear the camera's hardware logic supports concurrent writing to both cards, as needed to support the existing dual card modes.

Implementation would be trivial but my guess is Canon/Nikon would limit it to pro bodies since those are primarily differentiated by their buffer capacity.
 
Interesting, but just because it can write the same image to two cards simultaneously doesn't mean it can write two different images (to different cards) simultaneously. If it can't, you can't get your 2x improvement.

I can imagine a RAID implementation with three cards, when the performance demands are severe, (4K video, 3D video), but probably not for a few generations yet.
 
pho64 said:
Interesting, but just because it can write the same image to two cards simultaneously doesn't mean it can write two different images (to different cards) simultaneously. If it can't, you can't get your 2x improvement.

I can imagine a RAID implementation with three cards, when the performance demands are severe, (4K video, 3D video), but probably not for a few generations yet.
Click to expand...
It's already writing one image to one and another image to another in RAW-JPEG mode.
 
laspariahs said:
It's already writing one image to one and another image to another in RAW-JPEG mode.
Click to expand...
Exactly. Even though the RAW and JPEG are derived from the same capture, they represent different pieces of data in disparate areas of DRAM, both of which are written in parallel to each card. In terms of DMA concurrency it would be the same as writing two uncorrelated RAWs or two uncorrelated JPEGs.
 
Toward the same goal of increasing buffer drain throughput (and in turn buffer capacity), how about an interframe compression scheme? Right now the bodies support intraframe raw compression, which doesn't take advantage of the temporal and spatial relationships across frames. If you're shooting a continuous burst it's likely that the majority of the frames share common data which can be correlated via delta compression. Think football game where only the position of the player changes across frames, with the background either constant or offset by some factor based on framing movement.

The space savings from the delta frames would be enormous, creating much smaller parcels of data that need to be written to flash. This would create orders of magnitude increases in images per second throughput to the cards and in turn the effective buffer capacity.

I'm not sure if current Digic/Expeed processors would support this compression but since they already support it for their H.264 logic for video it's likely that they do. The compression would put extra processing time/bandwidth demands on the chips, maybe more than the video demands since no fields could be skipped in full-image compression like they are for video. Might require a generational spin of the chips to be implemented properly.
 
Continued...doing the interframe compression would require keeping the previous 'x' number of frames around in memory to feed the delta compression. But since the flash writes are slow relative to the compression time, you could be calculating the delta on the base frames at the same time they're being written to flash, thus the total pressure on buffer memory would be about the same.
 
Horshack said:
This has probably been thought of before but just in case...

Pro bodies and now more recently prosumer bodies (D7000) have dual card slots. Camera firmware provides different options on how the two cards are utilized:
  • Redundancy (write images to both cards)
  • Dedicated purpose (raw on one, jpeg on other)
  • Capacity aggregation (when one card fills up, camera siwtches/overflows to the second card).
How about a parallel distribution option, where every image alternates to the other card, producing a 2x increase in buffer drain throughput and thus an effective increase in buffer capacity? Originally I had thought RAID 0 to stripe each image across the two cards but since fast buffer drain is only needed for continuous shooting it's simpler to just distribute entire images across the cards. Plus there's inherent redundancy in alternating on an entire-image basis as it would likely be used for continuous shooting and so even if a card fails you'll have every other image on the non-failing card from the same shooting sequence.

Performance scaling would be 2x when the cards of equal speeds, even if one is CF and the other SD. If there is a speed differential between cards then firmware could alter the image ratio accordingly to balance the throughput. This could be done on a dynamic basis to accommodate asymmetry in card performance resulting from:
  • Base throughput differences between the cards
  • Different image sizes (worst-case distribution of image sizes in a sequence)
  • Outlier latencies on an individual card due to internal flash operations
Regarding hardware support for this, I don't think there's an internal bandwidth or DMA serialization issue since it's clear the camera's hardware logic supports concurrent writing to both cards, as needed to support the existing dual card modes.

Implementation would be trivial but my guess is Canon/Nikon would limit it to pro bodies since those are primarily differentiated by their buffer capacity.
Click to expand...
I was having these thoughts myself.
I would be a very excellent idea.

I would suggest that the camera writes the next image to whichever cardslot is ready though. If your cards are same speed you will get more or less what you say. If not, you will still get as much speed-gain as you can. Of course this means that if you put in a slow 64 GB card and a fast 32 GB card, before long the 32 GB card is full and the 64 GB only have perhaps 10 or 20 GB on it.

If you want to make it even more fancy you can ask how to split:
  • alternate images (even images to slot 1, odd images to slot 2)
  • First ready slot - to get max. possible speed gain (as described above)
  • According to capacity - so both slots fills up at the same time, i.e. you can shoot as long as possible with (at least some) speed gain
And then of course the options already there
  • Redundancy (everything on both)
  • RAW/JPG split or image/video split
  • Sequential (fill up slot 1 first, then slot 2)
 
Horshack said:
Continued...doing the interframe compression would require keeping the previous 'x' number of frames around in memory to feed the delta compression. But since the flash writes are slow relative to the compression time, you could be calculating the delta on the base frames at the same time they're being written to flash, thus the total pressure on buffer memory would be about the same.
Click to expand...
I don't think that will be successful though. There will be small changes all over the image, - even if you shoot at a tripod and shoot continuously. True H264 does something along those lines, - but it's not loss less. I don't think I want to loose quality for this.

If on the other hand you are able to do it. You should be able to increase not only the number of images in the buffer, but perhaps even the FPS (depending on where the bottleneck is and where you can do the compression).

But as said, I don't believe this will work.
 
DaDane said:
I would suggest that the camera writes the next image to whichever cardslot is ready though. If your cards are same speed you will get more or less what you say. If not, you will still get as much speed-gain as you can. Of course this means that if you put in a slow 64 GB card and a fast 32 GB card, before long the 32 GB card is full and the 64 GB only have perhaps 10 or 20 GB on it.
Click to expand...
This would be implicit to the balancing scheme. Once the first pair of image writes have started neither card would be ready (idle) in that sense, and in continuous sequence it's unlikely either card will ever be idle. Thus the balancing would require more precise throughput sampling, which could be achieved by simply establishing and tracking buffer watermarks for each card.
 
back at the computer? Three readers? Special software? Seems far easier to me to just increase in-card performance than the trouble it will take to implement RAID on multiple cards.

Think of how hard it is today on a desktop setup where usually a dead config means total loss of data. I think it is far too dangerous for "keeping" your photos safe. As long as there is a reason to write to the cards very quickly, a tiered approach may be best.

Camera can incorporate a much larger RAM or SRAM secondary buffer used to transition files from primary buffer in order to have a deeper buffer and still write to standard high speed cards on external card slots. No more complexity than today with added benefit. I think the D4 and D800 already have a ton of ram for this. I mean, the D4 beats my BUFFER UPGRADED D3 by a LARGE margin by DEFAULT!!! I get about 36 FX 12MP frames at 11fps today and am already salivating at getting more than DOUBLE this at 16MP!!

Shooting a fast paced Airshow is that demanding... no time to wait for the buffers to clear. Sometimes I have three 15 shot bursts back to back... so I will miss the third pass. With a deeper buffer, I can shoot more sets/passes before the buffer needs to empty.

To those who say 36 image buffer or 12MP is enough, they don;t realize the extra flexibility and freedom gained from 16MP and 100+ buffer :-)
--
Manny
http://www.pbase.com/gonzalu/
http://www.thrustimages.com/
FCAS Member - http://fcasmembers.com/
 
gonzalu said:
back at the computer? Three readers? Special software? Seems far easier to me to just increase in-card performance than the trouble it will take to implement RAID on multiple cards.
Click to expand...
It's not a stripe set. Each card would hold full images with unique filenames. The user would transfer them to his computer using the same workflow, either by reading the cards over the camera's USB/ethernet interface or via an external card reader.
gonzalu said:
Camera can incorporate a much larger RAM or SRAM secondary buffer used to transition files from primary buffer in order to have a deeper buffer and still write to standard high speed cards on external card slots. No more complexity than today with added benefit. I think the D4 and D800 already have a ton of ram for this. I mean, the D4 beats my BUFFER UPGRADED D3 by a LARGE margin by DEFAULT!!! I get about 36 FX 12MP frames at 11fps today and am already salivating at getting more than DOUBLE this at 16MP!!
Click to expand...
I suspect they would have already done this if it didn't have consequences, since DRAM part costs are relatively cheap for the price bands these cameras play in. It's likely they haven't done so due to the power requirements of the extra DRAM, and maybe to a lesser extent space requirements as well.
 
Horshack said:
DaDane said:
I would suggest that the camera writes the next image to whichever cardslot is ready though. If your cards are same speed you will get more or less what you say. If not, you will still get as much speed-gain as you can. Of course this means that if you put in a slow 64 GB card and a fast 32 GB card, before long the 32 GB card is full and the 64 GB only have perhaps 10 or 20 GB on it.
Click to expand...
This would be implicit to the balancing scheme. Once the first pair of image writes have started neither card would be ready (idle) in that sense, and in continuous sequence it's unlikely either card will ever be idle. Thus the balancing would require more precise throughput sampling, which could be achieved by simply establishing and tracking buffer watermarks for each card.
Click to expand...
The I misunderstood you. I though you wanted odd numbered pictures on one card and even numbered pictures on the other cards.

I there's a big difference in the speed then slot 1 can write image 1 and 3 while slot 2 only wrote image 2. That's o.k. but the slot 1 writes 5 and 7 and slot 7 just number 4. AFter a while slot 1 will soon run out of work because only odd even numbered pictures are left in the buffer.
 
actually I think 4k will kill the sports flagships,
that the 1DXI and D5 may become 4k video cameras,
hopefully with phase detection AF (on sensor or via a translucent mirror),
before the 2016 Rio de Janeiro Olymics and we may see 8k in 2020.
 
danharperphotography.com said:
That's really interesting Horshack. You come up with some really interesting stuff. Are you a software developer?
Click to expand...
Or perhaps he saw my post and ran with it. ;)
http://forums.dpreview.com/forums/read.asp?forum=1032&message=40906232

He certainly elaborated a lot more than I did! I like the alternating idea for its mix of capacity and redundancy. At most, you lose half your images if one card goes bad. Not as good as having complete redundancy, but much better than having none at all. I think it's a good compromise for certain missions.
danharperphotography.com said:
--
-Dan
Winnipeg, Manitoba, Canada
'Cameras don't take pictures, people do.'
'No one sees your camera when they're looking at your pictures.'
http://www.danharperphotography.com/ -BLOG/stock site
http://www.danharperphoto.com/ -Commercial portfolio
http://www.wpgphoto.com/ -My Winnipeg based photography community
Click to expand...
--
http://models.stevemelvin.com
 
XQD card is a simple answer only it's not popular now.
and for fun we may want to have a magazine of SDs for RAID-5.
 
stephenmelvin said:
Or perhaps he saw my post and ran with it. ;)
http://forums.dpreview.com/forums/read.asp?forum=1032&message=40906232

He certainly elaborated a lot more than I did! I like the alternating idea for its mix of capacity and redundancy. At most, you lose half your images if one card goes bad. Not as good as having complete redundancy, but much better than having none at all. I think it's a good compromise for certain missions.
Click to expand...
Check the timestamps of our messages ;-)
 
Horshack said:
stephenmelvin said:
Or perhaps he saw my post and ran with it. ;)
http://forums.dpreview.com/forums/read.asp?forum=1032&message=40906232

He certainly elaborated a lot more than I did! I like the alternating idea for its mix of capacity and redundancy. At most, you lose half your images if one card goes bad. Not as good as having complete redundancy, but much better than having none at all. I think it's a good compromise for certain missions.
Click to expand...
Check the timestamps of our messages ;-)
Click to expand...
Ha! Well there's a coincidence, isn't it? All I saw was "3 hours ago" for both of our posts, until I looked a bit closer. ;)

--
http://models.stevemelvin.com
 
You must log in or register to reply here.

Keyboard shortcuts

f
Forum
m
My threads
Mobile site
About
Editorial content
Cameras & Lenses
Community
All content, design, and layout are Copyright © 1998–2025 Digital Photography Review All Rights Reserved.
Reproduction in whole or part in any form or medium without specific written permission is prohibited.
When you use DPReview links to buy products, the site may earn a commission.
©GPS Media - Guides, Products, Services.
Back
Top