Demonstration of X-Trans Under-the-hood RAW Spatial (Noise) Filtering

Some of us over in the Astro forum have been taking a close look at some Fuji camera sensor characteristics. For those of you who don't follow this, astrophotographers have been discovering under the hood RAW data filtering in many Sony sensor based cameras. The well known star eater problem has been found in some Sony & Nikon cameras. An algorithm is applied to find and remove hot pixels & it frequently mistakes stars for hot pixels. Fuji seems to be doing something different.

An avid astrophotographer asked about the performance of the X-T3 & this sparked a very revealing group examination. The reason I posted here is these findings may explain the waxy/wormy artifacts & RAW converter choice/settings frequently debated in this & other Fuji forum. A star field provides an excellent way to examine these questions.

We've found that Fuji applies filtering to X-trans RAW data in camera that slightly smooths images & produces some star color artifacts but no filtering is used in the Bayer sensor X-T100. This is easily seen by comparing X-T2/3 images w/ X-T100 images. The later is much sharper. Ironic isn't it?

For many/most users & for web display, the difference will not be noticeable.  And the filtering seems to only apply for longer than 5 sec. exposures.

The differences between RAW converters handling of X-trans is trivial compared to the difference between X-T100 & X-T2/3. There is no way to turn the filtering off & Fuji has never disclosed this.

This is also how we found that electronic shutter & continuous drive modes in Fuji cams bin the RAW data down from 14 to 12 bit.

For a comparison of astro image results look HERE.

For a deep dive into Fuji sensor analysis, look HERE.

Not looking to start any war about X-trans just hoping to clarify what's going on. Hope its helpful.

This happens because people are too lazy to use dark frame these days. For years it was never a problem for anyone, but then someone started complaining it takes too much time, and I guess manufacturers are now trying to find ways to do it ”on the fly”, with these issues as the result. Note that dark frame in itself never eats stars, and can always tell the difference between a point of light and a hot pixel. Just goes to show that taking shortcuts never pays. Take the time to do the job right, you’ll get the best result in the end.

Kemist wrote:

tradesmith45 wrote:

For a deep dive into Fuji sensor analysis, look HERE.

This link doesn't work.

Thanks for letting me know.  Should have known this would happen, DPR was doing an update to the site as I was posting.

THIS should work.

razorfish wrote:

This happens because people are too lazy to use dark frame these days. For years it was never a problem for anyone, but then someone started complaining it takes too much time, and I guess manufacturers are now trying to find ways to do it ”on the fly”, with these issues as the result. Note that dark frame in itself never eats stars, and can always tell the difference between a point of light and a hot pixel. Just goes to show that taking shortcuts never pays. Take the time to do the job right, you’ll get the best result in the end.

Actually dark frames are a poor solution because they often add noise. This happens for several reasons. One significant problem is matching sensor temps between lights & darks. During a long shoot, the sensor warms & can take 2-3 hrs to reach equilibrium. Creating darks to match that is impractical. Applying darks that are from a different temp adds noise. Some cameras report sensor temps in the EXIF but many don't.  For the cameras that report sensor temp, it is theoretically possible to match temps but not easy.

A better solution for hot/stuck/dead pixels than star-eating is available in some camera specific RAW converters. A bad pixel map is used by the converter to ID which pixels need to have interpreted values applied.  I believe some astro specific software will also do this.

Fuji & Oly have a bad pixel in-camera mapping function for some models but IMHO, they don't work very well.

rlx wrote:

tradesmith45 wrote:

...

This is also how we found that electronic shutter & continuous drive modes in Fuji cams bin the RAW data down from 14 to 12 bit.

...

Thanks for pointing this out and good to know.

Have you any information about the electronic first curtain shutter mode? Is the data also stripped to 12-bit in that mode?

Fortunately E-first curtain is 14 bit - yeah!

That's a bit odd to use a 14-bit container file to store 12-bit data unless raw file standards are not flexible at all. I have no problem getting only 12 bit when using ISO values larger than 320 on the XT3 since the engineering dynamic range is always below 12 bits anyway at higher ISO values, but that's a loss of almost almost one stop at ISO160 though.

Yes, this will only matter is specific situations.  I've had them - back lit leopard hair in harsh African daylight @ base ISO.

One way you can see the change to 12 bit is the files are a bit smaller.

Thanks for posting this. Very interesting study. I can clearly see the importance of this for serious and dedicated astrophotographers. Looks like it may cause some to re-evaluate their equipment selection. Not sure about the implications for more casual "astrophotographers"...Too bad that the users have no control over the filtering.

jacob

jm10 wrote:

Thanks for posting this. Very interesting study. I can clearly see the importance of this for serious and dedicated astrophotographers. Looks like it may cause some to re-evaluate their equipment selection. Not sure about the implications for more casual "astrophotographers"...Too bad that the users have no control over the filtering.

jacob

Hope it helps some.  I had a couple other reasons for posting this.  I've seen hours processing my images thinking I was doing something wrong or that I'd gotten bad copies of lenses that was causing the muddy stars I got from the X-T2.  But it was the filtering.

Makes me PO'd that Fuji hasn't owned up to this & that's the another reason for the post.  Sony has acknowledged what they do.

tradesmith45 wrote:

jm10 wrote:

Thanks for posting this. Very interesting study. I can clearly see the importance of this for serious and dedicated astrophotographers. Looks like it may cause some to re-evaluate their equipment selection. Not sure about the implications for more casual "astrophotographers"...Too bad that the users have no control over the filtering.

jacob

Hope it helps some. I had a couple other reasons for posting this. I've seen hours processing my images thinking I was doing something wrong or that I'd gotten bad copies of lenses that was causing the muddy stars I got from the X-T2. But it was the filtering.

Makes me PO'd that Fuji hasn't owned up to this & that's the another reason for the post. Sony has acknowledged what they do.

Couple of things. First, do you think that this type of filtering has an effect on general photography (other than astrophotography) and secondly, could Fujifilm implement some sort of a "switch" if they wanted to?

jm10 wrote:

tradesmith45 wrote:

jm10 wrote:

Thanks for posting this. Very interesting study. I can clearly see the importance of this for serious and dedicated astrophotographers. Looks like it may cause some to re-evaluate their equipment selection. Not sure about the implications for more casual "astrophotographers"...Too bad that the users have no control over the filtering.

jacob

Hope it helps some. I had a couple other reasons for posting this. I've seen hours processing my images thinking I was doing something wrong or that I'd gotten bad copies of lenses that was causing the muddy stars I got from the X-T2. But it was the filtering.

Makes me PO'd that Fuji hasn't owned up to this & that's the another reason for the post. Sony has acknowledged what they do.

Couple of things. First, do you think that this type of filtering has an effect on general photography (other than astrophotography) and secondly, could Fujifilm implement some sort of a "switch" if they wanted to?

In the original post it is mentioned this is only applied to exposures of 5 seconds or longer

kiwidad wrote:

jm10 wrote:

tradesmith45 wrote:

jm10 wrote:

Thanks for posting this. Very interesting study. I can clearly see the importance of this for serious and dedicated astrophotographers. Looks like it may cause some to re-evaluate their equipment selection. Not sure about the implications for more casual "astrophotographers"...Too bad that the users have no control over the filtering.

jacob

Hope it helps some. I had a couple other reasons for posting this. I've seen hours processing my images thinking I was doing something wrong or that I'd gotten bad copies of lenses that was causing the muddy stars I got from the X-T2. But it was the filtering.

Makes me PO'd that Fuji hasn't owned up to this & that's the another reason for the post. Sony has acknowledged what they do.

Couple of things. First, do you think that this type of filtering has an effect on general photography (other than astrophotography) and secondly, could Fujifilm implement some sort of a "switch" if they wanted to?

In the original post it is mentioned this is only applied to exposures of 5 seconds or longer

Fair point but long exposures are used not only in astrophotography...

jm10 wrote:

Couple of things. First, do you think that this type of filtering has an effect on general photography (other than astrophotography) and secondly, could Fujifilm implement some sort of a "switch" if they wanted to?

Good questions. On the second one, don't know but they probably can implement a selection in firmware - if there's processor memory capacity available for the code. Sony changed their star-eater filter in a firmware update.

Regarding general photography, the answer broadly is the filtering will impact all RAW images w/ exposures over 5 sec. The filtering seems to impact all ISOs.

Specifically how it may effect general photography is harder to be exact about. One of the astro folks who worked on this & who has lots of experience w/ multiple camera brand filtering stated that unlike other sensors, he is unable to predict how images will be effected by Fuji's filtering based on his analysis. That is one of the big reasons I did the real astro image comparisons. Lets see what we might guess based on one of the image comparisons I posted:

The filtering is different in the 2 X-trans cameras but has the strongest impact on darker pixels in both. But even the brightest stars have been affected by smoothing in both. Whether the false color pixels in the X-T2 is the result of filtering or black points in each channel or something else is unknown.

So for general photography beyond 5 sec, we could expect filtering to slightly reduce shadow detail & for some X-trans give a color tint in deep shadows. But there could be some small reduction of all detail as would occur from increased NR adjustments in RAW conversion.

Careful testing by someone could look for this & confirm/disprove these expectations. A scene w/ a wide range of brightness w/ deep shadows, plenty of small detail, a sharp lens & several cameras would be needed.

Most RAW images will have NR applied during post. It may be that a useful strategy for dealing w/ the filtering would be to apply less NR in post for images over 5 sec. And sharpening methods & settings may need to be different if the filtering creates artifacts. I saw some evidence of artifacts but haven't tried to investigate that specifically.

Hope this helps.

tradesmith45 wrote:

jm10 wrote:

Couple of things. First, do you think that this type of filtering has an effect on general photography (other than astrophotography) and secondly, could Fujifilm implement some sort of a "switch" if they wanted to?

Good questions. On the second one, don't know but they probably can implement a selection in firmware - if there's processor memory capacity available for the code. Sony changed their star-eater filter in a firmware update.

Regarding general photography, the answer broadly is the filtering will impact all RAW images w/ exposures over 5 sec. The filtering seems to impact all ISOs.

Specifically how it may effect general photography is harder to be exact about. One of the astro folks who worked on this & who has lots of experience w/ multiple camera brand filtering stated that unlike other sensors, he is unable to predict how images will be effected by Fuji's filtering based on his analysis. That is one of the big reasons I did the real astro image comparisons. Lets see what we might guess based on one of the image comparisons I posted:

The filtering is different in the 2 X-trans cameras but has the strongest impact on darker pixels in both. But even the brightest stars have been affected by smoothing in both. Whether the false color pixels in the X-T2 is the result of filtering or black points in each channel or something else is unknown.

So for general photography beyond 5 sec, we could expect filtering to slightly reduce shadow detail & for some X-trans give a color tint in deep shadows. But there could be some small reduction of all detail as would occur from increased NR adjustments in RAW conversion.

Careful testing by someone could look for this & confirm/disprove these expectations. A scene w/ a wide range of brightness w/ deep shadows, plenty of small detail, a sharp lens & several cameras would be needed.

Most RAW images will have NR applied during post. It may be that a useful strategy for dealing w/ the filtering would be to apply less NR in post for images over 5 sec. And sharpening methods & settings may need to be different if the filtering creates artifacts. I saw some evidence of artifacts but haven't tried to investigate that specifically.

Hope this helps.

Thanks for your detailed response tradesmith. Lots of food for thought here...I was not aware of the false color pixels in X-T2 - looks quite obvious in your comparison. I would venture a guess that for most Fujifilm users your study may appear purely academic. I find it quite useful - we need to know the limitations and perhaps peculiarities of the tools we are using. It the very minimum there may be situations where a known camera weakness could be avoided.

In this case a single shot >5 sec in a deep shadow scene could be replaced with a combined couple of shots with a 2 or 3 sec exposure. This is of course if you are really concerned about the results:-)

Please keep us posted if you uncover more useful info.

jm10 wrote:

kiwidad wrote:

jm10 wrote:

tradesmith45 wrote:

jm10 wrote:

Thanks for posting this. Very interesting study. I can clearly see the importance of this for serious and dedicated astrophotographers. Looks like it may cause some to re-evaluate their equipment selection. Not sure about the implications for more casual "astrophotographers"...Too bad that the users have no control over the filtering.

jacob

Hope it helps some. I had a couple other reasons for posting this. I've seen hours processing my images thinking I was doing something wrong or that I'd gotten bad copies of lenses that was causing the muddy stars I got from the X-T2. But it was the filtering.

Makes me PO'd that Fuji hasn't owned up to this & that's the another reason for the post. Sony has acknowledged what they do.

Couple of things. First, do you think that this type of filtering has an effect on general photography (other than astrophotography) and secondly, could Fujifilm implement some sort of a "switch" if they wanted to?

In the original post it is mentioned this is only applied to exposures of 5 seconds or longer

Fair point but long exposures are used not only in astrophotography...

True but in your post you stated "other than astrophotography" and that is already answered by original post ( I may=je a bad assumption perhaps that general photography isn't long exposures...

, as far as 2 goes who knows.

Its interesting though since this whole thread seems to indicate Fuji do stuff to the raw data which may explain why the lesser cameras with Bayer seem to have sharper raw images straight out of the camera.

Has anyone tried first-gen X bodies like the X-Pro1 or X-E1? It would be interesting to see, as the whole fuss about high ISO skin smoothing started with 2nd-gen, and that was buried in the system.

57LowRider wrote:

Has anyone tried first-gen X bodies like the X-Pro1 or X-E1? It would be interesting to see, as the whole fuss about high ISO skin smoothing started with 2nd-gen, and that was buried in the system.

In our original post, we looked briefly at the XT10 & found its dark frame histogram was truncated indicating it is also filtered.

Before buying my 1st Fuji, I collected dark frames from several models & rented an X-Pro2. The dark frame histograms from all show signs of filtering but it differs by model. Heres the X-M1 & X-Pro2:

X-M1

X-Pro2

The X-Pro2 looks like the XT2. You have to look closely for the small steps in the histogram. Dark noise is random & should have no pattern. The XM1 is much more obvious & like the XT10. (From the recent tests, the X-T2 & XPro2 appear to also slightly clip blacks as well as apply spatial filtering to reduce long exposure noise. The XT3 does not appear to clip blacks.)

At the time I was first trying Fuji, I didn't have RawDigger nor understand any of this.

My early untracked star field images from the XT10 & XPro2 showed the characteristic missing bright cores from dim stars caused by filtering but I mistakenly chalked that up to lack of tracking & lens blur. Bright star cores are visible down to a low brightness level & then disappear for all similar stars. Weak colors are affected too.  This is definitely subtle! Star images from the X-T100 don't do this.

XT10

X-Pro2

It was only after I got a tracker, stopped my lenses down a bit & was still frustrated w/ muddy star fields that I began to dig deeper & learned of the filtering.

FWIW some Nikons apply filtering to exposures beyond 1/4 second. At least Fuji waits til 5 sec.

As for waxy skin & green foliage, these tests do not reveal filter for short exposures but it is very hard to detect filtering generally.

kiwidad wrote:

jm10 wrote:

kiwidad wrote:

jm10 wrote:

tradesmith45 wrote:

jm10 wrote:

Thanks for posting this. Very interesting study. I can clearly see the importance of this for serious and dedicated astrophotographers. Looks like it may cause some to re-evaluate their equipment selection. Not sure about the implications for more casual "astrophotographers"...Too bad that the users have no control over the filtering.

jacob

Hope it helps some. I had a couple other reasons for posting this. I've seen hours processing my images thinking I was doing something wrong or that I'd gotten bad copies of lenses that was causing the muddy stars I got from the X-T2. But it was the filtering.

Makes me PO'd that Fuji hasn't owned up to this & that's the another reason for the post. Sony has acknowledged what they do.

Couple of things. First, do you think that this type of filtering has an effect on general photography (other than astrophotography) and secondly, could Fujifilm implement some sort of a "switch" if they wanted to?

In the original post it is mentioned this is only applied to exposures of 5 seconds or longer

Fair point but long exposures are used not only in astrophotography...

True but in your post you stated "other than astrophotography" and that is already answered by original post ( I may=je a bad assumption perhaps that general photography isn't long exposures...

I guess the definition of what "general photography" is is in the eyes of the beholder...:-) I would not argue this point...

, as far as 2 goes who knows.

Its interesting though since this whole thread seems to indicate Fuji do stuff to the raw data which may explain why the lesser cameras with Bayer seem to have sharper raw images straight out of the camera.

There seems to be lots of opinions on this subject. I don't have a X-T100 to offer any wisdom. Not a big fan of JPEG images out of a camera (you don't see a raw image - it is a file consisting of bunch of ones and zeroes...)

To perform a  simple comparison of Bayer vs. X-Trans within an Adobe tool would not be fair. While X-T100 will perform well, an X-Trans camera may not and a tool with better demosaicking capabilities may be in order (like Capture One or PhotoNinja for example). Has been done before I think, but I have not payed enough attention since I was quite happy with the X-T2 performance.

Again, the opinion above would be applicable to so-called "general photography" - for astrophotography tradesmith45 presented convincing arguments that a Bayer sensor equipped camera may have an edge.

jm10 wrote:

kiwidad wrote:

jm10 wrote:

kiwidad wrote:

jm10 wrote:

tradesmith45 wrote:

jm10 wrote:

Thanks for posting this. Very interesting study. I can clearly see the importance of this for serious and dedicated astrophotographers. Looks like it may cause some to re-evaluate their equipment selection. Not sure about the implications for more casual "astrophotographers"...Too bad that the users have no control over the filtering.

jacob

Hope it helps some. I had a couple other reasons for posting this. I've seen hours processing my images thinking I was doing something wrong or that I'd gotten bad copies of lenses that was causing the muddy stars I got from the X-T2. But it was the filtering.

Makes me PO'd that Fuji hasn't owned up to this & that's the another reason for the post. Sony has acknowledged what they do.

Couple of things. First, do you think that this type of filtering has an effect on general photography (other than astrophotography) and secondly, could Fujifilm implement some sort of a "switch" if they wanted to?

In the original post it is mentioned this is only applied to exposures of 5 seconds or longer

Fair point but long exposures are used not only in astrophotography...

True but in your post you stated "other than astrophotography" and that is already answered by original post ( I may=je a bad assumption perhaps that general photography isn't long exposures...

I guess the definition of what "general photography" is is in the eyes of the beholder...:-) I would not argue this point...

, as far as 2 goes who knows.

Its interesting though since this whole thread seems to indicate Fuji do stuff to the raw data which may explain why the lesser cameras with Bayer seem to have sharper raw images straight out of the camera.

There seems to be lots of opinions on this subject. I don't have a X-T100 to offer any wisdom. Not a big fan of JPEG images out of a camera (you don't see a raw image - it is a file consisting of bunch of ones and zeroes...)

To perform a simple comparison of Bayer vs. X-Trans within an Adobe tool would not be fair. While X-T100 will perform well, an X-Trans camera may not and a tool with better demosaicking capabilities may be in order (like Capture One or PhotoNinja for example). Has been done before I think, but I have not payed enough attention since I was quite happy with the X-T2 performance.

Again, the opinion above would be applicable to so-called "general photography" - for astrophotography tradesmith45 presented convincing arguments that a Bayer sensor equipped camera may have an edge.

To be clear Jim, what I'm saying is NOT that the XT100 is better because it's Bayer but rather because its unfiltered. We don't know how an unfiltered X-trans would perform for long exposures because there aren't any.

Also the difference shows regardless of what software is used for RAW conversion.

tradesmith45 wrote:

jm10 wrote:

kiwidad wrote:

jm10 wrote:

kiwidad wrote:

jm10 wrote:

tradesmith45 wrote:

jm10 wrote:

Thanks for posting this. Very interesting study. I can clearly see the importance of this for serious and dedicated astrophotographers. Looks like it may cause some to re-evaluate their equipment selection. Not sure about the implications for more casual "astrophotographers"...Too bad that the users have no control over the filtering.

jacob

Hope it helps some. I had a couple other reasons for posting this. I've seen hours processing my images thinking I was doing something wrong or that I'd gotten bad copies of lenses that was causing the muddy stars I got from the X-T2. But it was the filtering.

Makes me PO'd that Fuji hasn't owned up to this & that's the another reason for the post. Sony has acknowledged what they do.

Couple of things. First, do you think that this type of filtering has an effect on general photography (other than astrophotography) and secondly, could Fujifilm implement some sort of a "switch" if they wanted to?

In the original post it is mentioned this is only applied to exposures of 5 seconds or longer

Fair point but long exposures are used not only in astrophotography...

True but in your post you stated "other than astrophotography" and that is already answered by original post ( I may=je a bad assumption perhaps that general photography isn't long exposures...

I guess the definition of what "general photography" is is in the eyes of the beholder...:-) I would not argue this point...

, as far as 2 goes who knows.

Its interesting though since this whole thread seems to indicate Fuji do stuff to the raw data which may explain why the lesser cameras with Bayer seem to have sharper raw images straight out of the camera.

There seems to be lots of opinions on this subject. I don't have a X-T100 to offer any wisdom. Not a big fan of JPEG images out of a camera (you don't see a raw image - it is a file consisting of bunch of ones and zeroes...)

To perform a simple comparison of Bayer vs. X-Trans within an Adobe tool would not be fair. While X-T100 will perform well, an X-Trans camera may not and a tool with better demosaicking capabilities may be in order (like Capture One or PhotoNinja for example). Has been done before I think, but I have not payed enough attention since I was quite happy with the X-T2 performance.

Again, the opinion above would be applicable to so-called "general photography" - for astrophotography tradesmith45 presented convincing arguments that a Bayer sensor equipped camera may have an edge.

To be clear Jim, what I'm saying is NOT that the XT100 is better because it's Bayer but rather because its unfiltered. We don't know how an unfiltered X-trans would perform for long exposures because there aren't any.

I hope that we understand it the same way. X-T100 is using a Bayer CFA in a 2x2 pattern. X-Trans cameras are using an X-Trans CFA in a more randomized 6x6 patter. Both versions are not using additional Low-Pass (hardware) filters on top of the arrays. What tradesmith45 discovered is that X-T100 is not using extra filtering in software but some (or all) X-Trans cameras do (after 5 seconds of exposure).

Also the difference shows regardless of what software is used for RAW conversion.

Yes, at least for astrophotography applications. We are on the same page here...