Interesting discovery about buffer clearing on R7

Did you try unprotecting the images and erasing them rather than reformatting the card?

Another experiment to try is deleting them in your computer (not on the camera).  Both of these might help narrow down what's going on.

definedphotography wrote:

Wildkits wrote:

I don't protect images on card, but I wonder if reusing a card with old unprotected images would be similarly problematic?

Yes, it will be slow. This is why you should always start out with a freshly formatted card and not delete files from in-camera.

In my experience, it isn't.  I always delete files rather than reformatting each time (reformatting resets the folder number to 0000, I store my images on disk with the same directory/filename as the card, so resetting the folder number to 0000 would wreck everything, and having to create new folders each time would be a big hassle).  I've not seen this, but I don't protect images.  I gave Alastair a few other suggestions for experiments to try to better characterize things.

The camera has to work around any existing files, slowing down the write process.

That shouldn't really matter (it has to anyway, as soon as there are any images on the card).  It may be related to the protection mechanism.

Alastair Norcross wrote:

PhotosFlight wrote:

CamerEyes wrote:

PhotosFlight wrote:

Are both cards identical?

Also, a memory card is effectively a solid state drive. As there is no moving head performing the writes it would surprise me if the overhead of a fragmented write would take much longer.

I write JPGs to one drive and CRAW to the other. I rarely do long bursts even though I shoot sports but have not noticed a buffer clearing lag. I have thousands of images on each card.

If the question is for me and not the OP, the answer is yes. Prograde V90 64gb.

Sorry, it was for the OP but I goofed. It was clear you had dual V90s.

Yes, my two cards are identical. They are Sony Tough 128gb V60 UHSII cards. I have them write consecutively, though, not concurrently. On the occasion when I noticed a long buffer clearing time, the camera was writing to a card that still had just over 300 protected images on it. The other card was empty. If I'd been thinking clearly, I would have switched to the other card to see whether it made a difference. I don't know why it made such a difference that the card had a bunch of protected files on it, but it did. As I said, when I performed the test with the card with 500 (I had protected another 200 images by then) protected images on it, it took about a minute to get to the point where the camera could take any more images, and two minutes until the buffer cleared completely. Once I reformatted the card, those times went down to 4 and 6 seconds respectively. A massive difference. My guess was that it was the fragmenting that caused this, but I don't really know. All I do know is that the effect is real, whatever it is that causes it. Also, as I said, I got more shots in the burst (about 85 CRAW) after reformatting than before (about 55 CRAW).

Try Prograde SD cards, V90 series. I am confident it will solve your buffer challenges.

CamerEyes wrote:

Alastair Norcross wrote:

PhotosFlight wrote:

CamerEyes wrote:

PhotosFlight wrote:

Are both cards identical?

Also, a memory card is effectively a solid state drive. As there is no moving head performing the writes it would surprise me if the overhead of a fragmented write would take much longer.

I write JPGs to one drive and CRAW to the other. I rarely do long bursts even though I shoot sports but have not noticed a buffer clearing lag. I have thousands of images on each card.

If the question is for me and not the OP, the answer is yes. Prograde V90 64gb.

Sorry, it was for the OP but I goofed. It was clear you had dual V90s.

Yes, my two cards are identical. They are Sony Tough 128gb V60 UHSII cards. I have them write consecutively, though, not concurrently. On the occasion when I noticed a long buffer clearing time, the camera was writing to a card that still had just over 300 protected images on it. The other card was empty. If I'd been thinking clearly, I would have switched to the other card to see whether it made a difference. I don't know why it made such a difference that the card had a bunch of protected files on it, but it did. As I said, when I performed the test with the card with 500 (I had protected another 200 images by then) protected images on it, it took about a minute to get to the point where the camera could take any more images, and two minutes until the buffer cleared completely. Once I reformatted the card, those times went down to 4 and 6 seconds respectively. A massive difference. My guess was that it was the fragmenting that caused this, but I don't really know. All I do know is that the effect is real, whatever it is that causes it. Also, as I said, I got more shots in the burst (about 85 CRAW) after reformatting than before (about 55 CRAW).

Try Prograde SD cards, V90 series. I am confident it will solve your buffer challenges.

Well, I don't actually have any buffer challenges anymore. As I wrote, after reformatting my Sony Tough card, it took a whopping 4 seconds until it was ready to shoot again, and a total of 6 seconds to completely clear the buffer. Two Prograde SD V90 128gb cards would cost me $300. How many seconds would I save for that extra $300?

Robert Krawitz wrote:

Did you try unprotecting the images and erasing them rather than reformatting the card?

No. That's something to try. But if I'm going to erase them anyway, why not simply reformat the card? I know that works. I suppose this might satisfy our curiosity as to what causes the slowdown. But now that I know how to get the buffer to clear in 6 seconds, I think I'll stick to that technique.

Another experiment to try is deleting them in your computer (not on the camera). Both of these might help narrow down what's going on.

I could try that. But the whole point of protecting them in camera was so that I didn't have to download 3000 images to my computer, before erasing them. Or did you mean that I should attach the card to the computer, and delete them off the card that way? Without downloading all the ones I don't want to the computer?

Robert Krawitz wrote:

That shouldn't really matter (it has to anyway, as soon as there are any images on the card). It may be related to the protection mechanism.

That can and does matter, depending on the write algorithm. While it has to be done, the efficiency of doing so can change

Take for example if the write algorithm looks for first empty slot, then checks if X amount of contiguous space is available. If you start from an empty / formatted card, this condition will always be fulfilled on first try. As images get added sequentially, you can still find this on first attempt until the card is nearly full

Now try doing the same with files deleted in pockets. You might have to attempt more than once to fulfill the criteria. How conservative X is also matters, because if the writes are trying to avoid filesystem fragmentation without knowing exact size of file to be written upfront, they have to use some heuristic with padding to be effective

There can be more similar nuances. I think we are dealing with filesystem overheads here. Not ruling out the effect of protect itself, it probably creates its own data structure that might interfere further, and both aspects can be at play together too

Having said above, I am a bit surprised with the results myself. While I expect overheads to increase, I would have expected a smaller scale when dealing with flash storage

CamerEyes wrote:

The best solution to me for overcoming the R7's buffer limitations? Use two V90 Prograde SD cards. One card saving RAW files, the other saving JPEGs, and video being saved on the card where JPEGs are being saved.

Are the V90 Prograde cards immune to the slowdown shown in OPs experiment? Have you tested it?

Never had to worry about buffer anymore. Yes, it's a $250 solution, not particularly cheap.

If you confirm that similar slowdown doesn’t happen in same conditions, it may very well be a solution. But if it does, it may be an expensive way to discover that control matters in an experiment

Thanks for sharing this

Would be a good experiment if you took a partially filled card with protected images only, and a fresh formatted in camera, then took them both to a computer and tried to copy a lot of images of similar size on to them

If we see similar speed differences, its inherent to the card+filesystem. If not, it’s probably the write algorithm the camera uses

Alastair, have you tried to see if your R6ii behaves in the same way with the same SD cards?

Alastair Norcross wrote:

Robert Krawitz wrote:

Did you try unprotecting the images and erasing them rather than reformatting the card?

No. That's something to try. But if I'm going to erase them anyway, why not simply reformat the card? I know that works. I suppose this might satisfy our curiosity as to what causes the slowdown. But now that I know how to get the buffer to clear in 6 seconds, I think I'll stick to that technique.

I suggested it as an experiment to try to better characterize the behavior (I suppose I could try that too, since I have an R7).  I'm a software engineer, so I think in terms of debugging like that.

The reason I don't like to reformat my card, as I said, is that it resets the folder number.  Canon clearly stores the image number in the camera, I don't understand why the firmware doesn't also store the folder number.  I download all of my frames to my computer (Sandisk Extreme Pro mkII -- 300 MB/sec with a good reader), copy them to my other computer, and have a few backup sets, then just delete them all from the card.

Another experiment to try is deleting them in your computer (not on the camera). Both of these might help narrow down what's going on.

I could try that. But the whole point of protecting them in camera was so that I didn't have to download 3000 images to my computer, before erasing them. Or did you mean that I should attach the card to the computer, and delete them off the card that way? Without downloading all the ones I don't want to the computer?

I meant the latter (attach it to your computer and delete them there, using a file manager or command line, not DPP or some other Canon-specific tool).  Again, as an experiment.

PicPocket wrote:

Robert Krawitz wrote:

That shouldn't really matter (it has to anyway, as soon as there are any images on the card). It may be related to the protection mechanism.

That can and does matter, depending on the write algorithm. While it has to be done, the efficiency of doing so can change

Take for example if the write algorithm looks for first empty slot, then checks if X amount of contiguous space is available. If you start from an empty / formatted card, this condition will always be fulfilled on first try. As images get added sequentially, you can still find this on first attempt until the card is nearly full

Conceivably, but I'm pretty sure exFAT is designed a bit more intelligently, and someone would have to go far out of their way to muck the implementation up that badly.

Now try doing the same with files deleted in pockets. You might have to attempt more than once to fulfill the criteria. How conservative X is also matters, because if the writes are trying to avoid filesystem fragmentation without knowing exact size of file to be written upfront, they have to use some heuristic with padding to be effective

With flash memory there's little reason to worry about fragmentation, especially with files as big as images are.  JPEG, RAW, and CRAW files are all on the order of 10 MB, which are much bigger than the 4K or so block size of the filesystem.

There can be more similar nuances. I think we are dealing with filesystem overheads here. Not ruling out the effect of protect itself, it probably creates its own data structure that might interfere further, and both aspects can be at play together too

That's why I suggested deleting images from the card on a computer, and without using DPP, which might fix up the protection information itself (unless the protection is relying on file permissions).  I know that there are a few small files in another directory (CANONMSC, as I recall) that look like they could contain something like that.

Having said above, I am a bit surprised with the results myself. While I expect overheads to increase, I would have expected a smaller scale when dealing with flash storage

davesurrey wrote:

Alastair, have you tried to see if your R6ii behaves in the same way with the same SD cards?

Not yet, but I’ll have an opportunity this weekend, so maybe I’ll give it a try.

Some software developer at Canon knows why.  Very very strange.

Alastair Norcross wrote:

CamerEyes wrote:

Alastair Norcross wrote:

PhotosFlight wrote:

CamerEyes wrote:

PhotosFlight wrote:

Are both cards identical?

Also, a memory card is effectively a solid state drive. As there is no moving head performing the writes it would surprise me if the overhead of a fragmented write would take much longer.

I write JPGs to one drive and CRAW to the other. I rarely do long bursts even though I shoot sports but have not noticed a buffer clearing lag. I have thousands of images on each card.

If the question is for me and not the OP, the answer is yes. Prograde V90 64gb.

Sorry, it was for the OP but I goofed. It was clear you had dual V90s.

Yes, my two cards are identical. They are Sony Tough 128gb V60 UHSII cards. I have them write consecutively, though, not concurrently. On the occasion when I noticed a long buffer clearing time, the camera was writing to a card that still had just over 300 protected images on it. The other card was empty. If I'd been thinking clearly, I would have switched to the other card to see whether it made a difference. I don't know why it made such a difference that the card had a bunch of protected files on it, but it did. As I said, when I performed the test with the card with 500 (I had protected another 200 images by then) protected images on it, it took about a minute to get to the point where the camera could take any more images, and two minutes until the buffer cleared completely. Once I reformatted the card, those times went down to 4 and 6 seconds respectively. A massive difference. My guess was that it was the fragmenting that caused this, but I don't really know. All I do know is that the effect is real, whatever it is that causes it. Also, as I said, I got more shots in the burst (about 85 CRAW) after reformatting than before (about 55 CRAW).

Try Prograde SD cards, V90 series. I am confident it will solve your buffer challenges.

Well, I don't actually have any buffer challenges anymore. As I wrote, after reformatting my Sony Tough card, it took a whopping 4 seconds until it was ready to shoot again, and a total of 6 seconds to completely clear the buffer. Two Prograde SD V90 128gb cards would cost me $300. How many seconds would I save for that extra $300?

Good point. I have zero lag, no delay due buffering. Essentially I call that peace of mind I've made a decision to buy a third Prograde V90 SD card as a result (second for the R7).

Originally, I had one for my R6Mii and one for my R7. In the R6Mii, I didn't need to have two such SD cards since it has generous headway.

CamerEyes wrote:I

have zero lag, no delay due buffering.

I am not sure what you mean. Do you mean you are able to shoot RAW or CRAW indefinitely on R7 without a pause? Beyond the specified 187 shot limit too?

Thanks for the info.

PicPocket wrote:

CamerEyes wrote:I

have zero lag, no delay due buffering.

I am not sure what you mean. Do you mean you are able to shoot RAW or CRAW indefinitely on R7 without a pause? Beyond the specified 187 shot limit too?

RAW shooting almost indefinite. I don't get a lag until around 200-250 shots later. I was surprised myself.

CamerEyes wrote:

PicPocket wrote:

CamerEyes wrote:I

have zero lag, no delay due buffering.

I am not sure what you mean. Do you mean you are able to shoot RAW or CRAW indefinitely on R7 without a pause? Beyond the specified 187 shot limit too?

RAW shooting almost indefinite. I don't get a lag until around 200-250 shots later. I was surprised myself.

Is this at max fps (15 or 30)?

Robert Krawitz wrote:

CamerEyes wrote:

PicPocket wrote:

CamerEyes wrote:I

have zero lag, no delay due buffering.

I am not sure what you mean. Do you mean you are able to shoot RAW or CRAW indefinitely on R7 without a pause? Beyond the specified 187 shot limit too?

RAW shooting almost indefinite. I don't get a lag until around 200-250 shots later. I was surprised myself.

Is this at max fps (15 or 30)?

15fps on EFCS and 30fps on ES. Using the RF24mm and the EF100-400mm Mii.