Emergency: X-T3 recorded approx 6000 files, iMac only shows first 4096 files

Hiya, shot a wedding yesterday on my two X-T3s. Both had two 128GB cards loaded, set to 'backup' (so the second card is a duplicate of the first). Shot in Raw+JPG mode.

I shot around 80GB on each camera, around 3000 images, or 6000 files (raw + jpg)

When I review the photos on my cameras, all the files are there, on all the cards.

When I load the cards into my iMac and do 'Get Info' on the cards themselves (not the folders inside) I see the appropriate data usage.

But when I actually go to download the computer only finds the first 2048 images that I shot on each card - on ALL the cards!

In the Finder, the cards only have one DCIM folder each, and that folder has exactly 4096 files (2048 raws, 2048 JPGs).

The remaining images just can't be seen by the computer! Doesn't matter if I look using Lightroom or Photo Mechanic or just the Finder.

It's almost as if the cameras both started a new folder after recording those 2048 images, and on both cameras, and all four cards, the folder is missing.

Am currently scanning one of the cards with Image Rescue 4 to see if it can see the missing images.

But... what on earth happened? Anyone recognise this issue? There's just no way all FOUR of these SD cards all failed on the same day. And very unlikely both cameras failed on the same day too. Is this a bug?

Help!

I've found a VERY helpful thread here that suggests that certain computer systems won't show more than 4096 files on a memory card for some ridiculous reason:

https://www.dpreview.com/forums/post/65187965

This user mentions Linux. I'm on Mac OS 10.14.6 (Mojave) which as far as I can tell isn't based on or built on Linux, so that the **** is happening here?

It sounds like if I transfer by USB directly from the camera I can bypass this issue. But that's a terrible solution for me. I'm going to be shooting this many files at weddings all the time and my workflow speed is heavily dependant on putting all the cards in a multi-card reader and batch downloading at once.

I've only started using 128GB cards in the last two years but this is the first long wedding I've done since I got them, because of COVID, so this is the first time I've encountered this.

There's GOT to be a solution, surely, that doesn't involve me hooking up my cameras to the computer to download. Any ideas?

Can you see them in the camera? Maybe then hook up your camera to the computer?

I've never had that problem with the X-T3. I just copy everything on the card to the computer. Often there's more than one file folder on the card, but I just copy all the images in each folder.

macOS is based on Darwin which itself is based on BSD Unix. It shares some user space components with GNU Linux, but most of its tools are compiled from source specifically for Darwin and can, at times, be behind the versions available on Linux.

As far as I know Fuji cameras format memory cards with the exFat file system, which doesn’t suffer from the file size and count limitations of older implementations of FAT.  My guess is you used an SD card with a factory format rather than formatting it in your camera before use. In that case it could be formatted using FAT or FAT16 which could be giving you problems. Transferring the images with the card in the camera would probably work since the camera presents to the OS differently than a card plugged into a reader will.

Once you get the images off be sure to format the card in-camera before using it again.

sdickenson wrote:

My guess is you used an SD card with a factory format rather than formatting it in your camera before use. In that case it could be formatted using FAT or FAT16 which could be giving you problems. Transferring the images with the card in the camera would probably work since the camera presents to the OS differently than a card plugged into a reader will.

Once you get the images off be sure to format the card in-camera before using it again.

Hiya, nope, I format my cards in camera before every shoot! Literally, every shoot. These cards have been formatted dozens of times before, and were formatted again on Saturday morning before the wedding.

Since Fuji updated their firmware to allow 9999 images per folder I've not shot more than 2000-ish images per camera on a wedding, before (4000 files), so I've never experienced this problem. Before that they allowed 999 images per folder, so the issue would never have occurred as you need to have 4096 files in one folder to see the issue.

I've found at least two other threads on other Fuji forums describing this exact issue with Macs. So it doesn't sound like a freak occurrence - but even if it is, the fact it can happen is a massive issue, at least for me.

dual12 wrote:

I've never had that problem with the X-T3. I just copy everything on the card to the computer. Often there's more than one file folder on the card, but I just copy all the images in each folder.

I've been using since X-T1 and always remember seeing multiple folders per card but didn't they update in the last year or two to allow 9999 images per folder now? That introduces the potential for this issue. I've not seen it until now because I've not had any full day weddings since late 2019, and I only started using 128GB cards mid-2019 and I suspect I either didn't shoot this many images, or the firmware created more folder.

sir_c wrote:

Can you see them in the camera? Maybe then hook up your camera to the computer?

Yes, they're visible in camera. I rescued them via Image Capture, which could see them. But it's so slow, and only one camera at a time, it's not a viable long term solution.

owenb wrote:

I've found a VERY helpful thread here that suggests that certain computer systems won't show more than 4096 files on a memory card for some ridiculous reason:

https://www.dpreview.com/forums/post/65187965

This user mentions Linux. I'm on Mac OS 10.14.6 (Mojave) which as far as I can tell isn't based on or built on Linux, so that the **** is happening here?

MacOS is POSIX compliant Unix OS.  Linux is a knock of Unix.  For for all practical purposes, they are using very similar kernels.  I don't think that is the problem.  I have collect many more files testing a RF processing system on a Mac.  However, I was not using a Fuji camera with the cards formatted in the camera.  To get to the root of this one would need to know exactly how Fuji formats its cards.

I seriously doubt that it is an OS issue.  I have downloaded disk with over 100,000 files of RF collects into onto a Mac Disk after a drive test.

It could be a card issue.  It could be a camera issue with large cards.  I seriously doubt it is a Unix/Linux issue.

Hi all, Fujifilm UK have confirmed this morning that Japan is aware of this issue and it’s a Mac OS problem.

Mac OS cannot read more than 4096 files in a folder on an SD card. End of story. I was advised this affects all camera manufacturers, not just Fujifilm, and there’s nothing they can do to fix it themselves unfortunately.

Thats true, and not quite true. I’ve found that recent Canon and Nikon cameras both write >4096 files to their card folders. But the Sony A9 writes 4000 files per folder. Just under the limit that Mac OS will read, which suggests they know about the issue. Or just got lucky?

Fujifilm UK said they’ll pass these notes on to Japan so, maybe there’s something can be added in a firmware update to either reduce the limit to 4000, or allow us to manually activate that limit in-camera.

in the meantime, if you’re a Mac user be careful not to shoot more than 4096 images (or 2048 if you shoot raw+jpg) or you’ll need to use Image Capture to download your card in full…

In case anyone disagrees this is a Mac OS issue, here’s what Fujifilm UK said in a message to me:

”Hi Owen,
I have already received a reply from the tech team in Japan and they are aware of this problem but the limitation is a Mac OS one and there's nothing we can do. This is the same for all camera manufacturers, it's not an exclusive problem for FUJIFILM cameras. The maximum number of files they can recognise in a folder on an SD card is 4096.”

That should be amenable to a firmware fix whereby a new folder is created.

(OTOH my wedding (1976) was covered very adequately on a handful of rolls of 120 film at 16 frames per roll.)

a_c_skinner wrote:

That should be amenable to a firmware fix whereby a new folder is created.

(OTOH my wedding (1976) was covered very adequately on a handful of rolls of 120 film at 16 frames per roll.)

in very hopeful news, UK has advised that Japan is not unwilling to consider a 4000 limit. But that doesn’t mean anything, so I’m hopeful but not celebrating yet!

owenb wrote:

a_c_skinner wrote:

That should be amenable to a firmware fix whereby a new folder is created.

(OTOH my wedding (1976) was covered very adequately on a handful of rolls of 120 film at 16 frames per roll.)

in very hopeful news, UK has advised that Japan is not unwilling to consider a 4000 limit. But that doesn’t mean anything, so I’m hopeful but not celebrating yet!

Maybe its because you are using an old version of MacOS?

There is no problem with version 11.4.

Greybeard2017 wrote:

owenb wrote:

a_c_skinner wrote:

That should be amenable to a firmware fix whereby a new folder is created.

(OTOH my wedding (1976) was covered very adequately on a handful of rolls of 120 film at 16 frames per roll.)

in very hopeful news, UK has advised that Japan is not unwilling to consider a 4000 limit. But that doesn’t mean anything, so I’m hopeful but not celebrating yet!

Maybe its because you are using an old version of MacOS?

There is no problem with version 11.4.

Maybe, I'm on Mojave. I'm sure I've seen people say they've seen it with Big Sur. But like I say, Fujifilm Japan say they know that this is an issue, so... yeah.

Is that personal observation or from a reliable source?  On Intel and M1?  Not disagreeing just trying to expand a useful contribution.

webfrasse wrote:

owenb wrote:

Hi all, Fujifilm UK have confirmed this morning that Japan is aware of this issue and it’s a Mac OS problem.

Mac OS cannot read more than 4096 files in a folder on an SD card. End of story. I was advised this affects all camera manufacturers, not just Fujifilm, and there’s nothing they can do to fix it themselves unfortunately.

Thats true, and not quite true. I’ve found that recent Canon and Nikon cameras both write >4096 files to their card folders. But the Sony A9 writes 4000 files per folder. Just under the limit that Mac OS will read, which suggests they know about the issue. Or just got lucky?

Fujifilm UK said they’ll pass these notes on to Japan so, maybe there’s something can be added in a firmware update to either reduce the limit to 4000, or allow us to manually activate that limit in-camera.

in the meantime, if you’re a Mac user be careful not to shoot more than 4096 images (or 2048 if you shoot raw+jpg) or you’ll need to use Image Capture to download your card in full…

Image Capture is the default app for importing images into macOS. Why not use it? I would never use the Finder for stuff like this. I've been using Photo Mechanic for years though but when I used Aperture I'm pretty sure the import was done via Image Capture.

With Photo Mechanic you can let it do a whole lot of processing and organizing while importing. A huge time saver...

Image Capture is a terrible solution, I'm genuinely confused as to why you'd suggest it, sorry! Firstly, it takes several minutes just to read the card. Then it takes a considerable amount of time to download the images. Finally, you need to be there to disconnect the first camera and connect the second camera.

Conversely, I've been using either PM or LR for import for years, and both allow me to plug my cards into dual card readers and import everything with one click, including assigning metadata on import, set it going with one click and forget about it. 
LR is actually even faster than PM for importing, too, because of how it reads the cards differently. PM goes through the cards finding everything chronologically, but LR imports all the blocks as it finds them on a card (images are not written to the card in chronological order necessarily) which is quite a lot faster for a large wedding shoot.

a_c_skinner wrote:

Is that personal observation or from a reliable source? On Intel and M1? Not disagreeing just trying to expand a useful contribution.

Observation on M1 - I took a look when the original Linux problem was posted.

Finder, Terminal and Lightroom Classic can all see more than 4096 files in a single folder.