X10: No RAW converter with full EXR support?

When looking at the raw data - 'ISO DR' data will always 'appear' like underexposure, indistinguishable from the under-exposed half of the 'dual exposure-time DR' method.
Only when amplification/ISO of the whole sensor is decreased, not when only half is decreased.
Whether 'half' or 'whole' - I mean either some part, or all, will 'appear' under-exposed.
... It's a harder task to combine nominally differently exposed halves (counting Iso as image exposure) than combining equally exposed ones.
'Harder' , yes - but 'hard', no.

I really don't see any great difficulty in combining the two different ISO/exposure halves - the combination only needs to follow some very basic, and very straight-forward, logic and rules.
Both ISO based DR and EXPosure based are a combination of hardware underexposure + curves. It doesn't really make sense to call one hardware and the other software.
....Yes, indeed I mostly agree there - although it is sometimes necessary to use terms that others here are more familiar with.
But terms are I'll-chosen, because they judge one method as being "true" and the other as being "just a simple software trick".
The origin of the 'software' term is understandable though - because for a full resolution 'L' size image, in essence the camera may appear to be just doing what a user could do for themselves by under-exposing -EV and then lifting with a curve in post-processing 'software'.

Until now, I don't think many have considered the possibility of split-sensor ISO/gain - which necessitates 'internal hardware' functionality.
BTW, ISO DR 400 pulls down highlights by about -2.5 EV, not just 2 full stops. So it seems that after hardware ISO/amplification underexposure happened the highlights are further pulled down by curves (there is headroom left).
I'm not convinced - I wonder if you may be mistaken there. It's complicated/confusing to try to evaluate 'EV' within 'curved' data.
I meant in Jpg. Dr 400 stronger clipping/highlight protection than corresponding Dr 100 shots at 2 stops lower Iso accomplish, appr. 2.5 stops total (until clipping areas match).
I think you mean '2 stops lower exposure' (not "lower Iso" )?

Anyhow, I see what your saying - although I think it would be clearer/fairer to say that the DR100% setting appears to waste a fraction of a stop of highlight range, and that the DR400% doesn't, so the difference appears larger than just 2 stops - undoubtedly just a matter of differences in respective tone curves.
I agree it is possible that the 'ISO DR' could use two different ISO/amplification settings - although the actual 'exposure' is still the same (same shutter speed and aperture), so there is no "overexposed half" as such.
Yes, I used bad wording there. I only meant split operation and different outputs of both halves. That exposure of the photo sites to photons remains the same is understood (and the reason why Iso Dr cannot help against blooming induced orbs).
Whether 'ISO DR' helps mitigate against blooming/'orbs' depends on what comparison, or starting point, basis is being made...

E.g. Assuming 12MP 'L' size, and therefore only 'ISO DR' is available...

If 'blooming/orbs' occurs at 'ISO200 DR100%' then increasing the 'DR' to DR200% (the max for ISO200) certainly won't help, because sensor exposure doesn't change.

But - increasing the so-called 'DR' to DR400% demands that the ISO be increased to ISO400 for DR400%, thereby resulting in lower sensor exposure and thereby reduced 'blooming/orbs'.

Now of course, it actually the increased 'ISO' that is reducing the 'blooming/orbs', not the 'DR' as such - but the need for higher 'DR' dictates the higher 'ISO', so the two values can be considered to be 'symbiotic'.
 
Interesting, but - it seems to show more about the vagaries/inadequacies of different 'third party' raw converters, and doesn't reveal that all that much about the camera itself.
Some general observations about X10 RAF files that I learned from this comparison, like always these are my personal observations and interpretations:

OOC JPGs are lens (distortion) corrected!
That's no surprise.

Early sample wide-angle shots looked far too 'straight' - I thought they were quite obviously 'corrected' in camera by software/firmware.

Many cameras use lens distortion correction these days - it's a lot easier/cheaper to fix this kind of thing in software/firmware than it is to design and manufacture better optics.

I remember seeing an uncorrected wide-angle shot of a Canon S95, and the barrel distortion was quite shocking.

I would almost guarantee that the camera also deploys quite significant lens vignetting compensation too, when required, and just possibly even some chromatic aberration correction.
Both 12 MP and 6 MP images (DR 100-400) store all information from both sensor halves in the RAW file which then needs to be properly decoded and combined by the RAW software. This is where all tested RAW converters fail compared to OOC JPG!
'Raw' should always 'ideally' record/save all sensor data where ever possible - ideally allowing the user to change choices/options later.
Only 6 MP RAWs of 9 mb filesize already contain the combined sensor information and thus don't rely on the RAW software for decoding. With DR 200/400 the 9 mb files are already noise-reduced and offer full color saturation (see below) in all RAW converters that can open them.
As you probably remember - I've previously suggested that it's likely that the camera may switch to a combination of software/ISO DR with 'pixel-pair binning' at higher ISO, instead of 'dual/split-exposure DR'.
It also seems like ISO based DR indeed uses only half the sensor for ISO/amplification tricks, which then has to be properly combined with the overexposed half just like EXPosure time based DR.
I don't think that is likely at all - I don't see your reasoning.

When the camera doesn't use it's 'dual/split-exposure' method, then all pixels are exposed the same, so they should all be the same signal/exposure and all equally useful and valuable.
As Silkypix was the 'Raw translator' of choice by Fuji, I cannot see how it is designated "third party". Purely by my definition and as it's the supplied item it is the primary converter although made by a company other than Fuji. It is after all supplied by Fuji.

Dave. (UK)
 
Whether 'half' or 'whole' - I mean either some part, or all, will 'appear' under-exposed.
I am not so sure, though, that curves lifting exposure back are applied by the RAW converter, it may well be that they are applied in-camera before creating the RAW.

Anyways, the basic issue of how RAW converters fail has all to do with half vs. full. My observation is that all images that are sure to contain the same exposure on both halves work properly (except for the loss of detail at 12 MP vs. OOC JPG). All images where supposedly exposure differs between the halves is where RAW software fails to produce a good image (loss of saturation and noise increase).
'Harder' , yes - but 'hard', no.

I really don't see any great difficulty in combining the two different ISO/exposure halves - the combination only needs to follow some very basic, and very straight-forward, logic and rules.
Tell that to Adobe, Phase One and ISL. :P My guess is that they don't want to invest the manpower necessary to cover the whole EXR + DR range, while all other cams work with simple Bayer patterns.
Both ISO based DR and EXPosure based are a combination of hardware underexposure + curves. It doesn't really make sense to call one hardware and the other software.
I meant in Jpg. Dr 400 stronger clipping/highlight protection than corresponding Dr 100 shots at 2 stops lower Iso accomplish, appr. 2.5 stops total (until clipping areas match).
I think you mean '2 stops lower exposure' (not "lower Iso" )?
Both. I mean that a ISO 400 DR 400 JPG image saves about 0.5 stops more clipped highlights than a ISO 100 DR 100 JPG. Working on the RAW of the ISO 100 shot needs 2.5 stops exposure in order to see about the same clipping as the ISO 400 DR 400 JPG.

You can test it yourself. Point a bright (LED) flashlight towards a door-frame at some angle. Keep the door open and the background room dark. Shoot that at higher ISO + DR 400 and then turn down 2 stops of ISO + DR 100. The DR 400 shot will show less area clipping white from the flashlight than the 2 stops lower ISO shot. Open the RAW of the lower ISO + DR 100 shot and pull down exposure until you can match the clipping area with the DR 400 shot.
Anyhow, I see what your saying - although I think it would be clearer/fairer to say that the DR100% setting appears to waste a fraction of a stop of highlight range, and that the DR400% doesn't, so the difference appears larger than just 2 stops - undoubtedly just a matter of differences in respective tone curves.
Yes, you are right, else we could not pull exposure down further in the RAW. But for ooc JPGs it's good to know that you get that extra headroom by using ISO DR. I have yet to check how EXP DR compares to that.
If 'blooming/orbs' occurs at 'ISO200 DR100%' then increasing the 'DR' to DR200% (the max for ISO200) certainly won't help, because sensor exposure doesn't change.
Just what I said.
But - increasing the so-called 'DR' to DR400% demands that the ISO be increased to ISO400 for DR400%, thereby resulting in lower sensor exposure and thereby reduced 'blooming/orbs'.
For the sake of clarity: No, in two ways! Sorry. ;)

1) ISO DR reduces ISO by 1 or 2 stops, it does not increase ISO. The one and only setting where ISO is nominally increased happen when using A-perture priority mode at ISO below 400 and exposure times longer than 1/4 second. "Nominally" means that the same lifting curves are applied to ISO 100 shots that ISO 400 shots would use after decreasing ISO to 100 internally (at DR 400).

And ironically this very specific situation reveals a firmware bug - or rather user conflict - where the camera may not reduce exposure time to compensate for the nominally higher ISO. Metering will still use exposure time based on base ISO (without curves) and may thus considerably overexpose. This happens when ISO and DR are not set to AUTO.

What happens is that the camera/firmware doesn't know it has to decrease exposure time, because for ISO DR this normally is not done. Would you use AUTO ISO with DR then the camera would choose a minimum of ISO 400 and exposure time would change accordingly. Would you use AUTO DR with ISO 100 then the camera would use DR 100 for any exposure time longer than 1/4s.

This is why I wrote "user conflict" above, the user asks the camera to perform an action that is not really possible with its current programming that says: ISO DR does not change exposure time!

So ISO DR at ISO lower than 400 is an anomaly and only happens because of one reason: to allow the user to freely choose an aperture and not be limited by the max 1/4s EXP DR or min ISO 400 ISO DR limitations. All other modes impose those limits. In fact A-mode behaves erratic in other aspects, too, all only because of this special treatment (and the confusion of the firmware programmers).
Now of course, it actually the increased 'ISO' that is reducing the 'blooming/orbs', not the 'DR' as such - but the need for higher 'DR' dictates the higher 'ISO', so the two values can be considered to be 'symbiotic'.
2) If you are using any of the (semi)-automatic modes then the camera automatically decreases exposure time or closes the aperture to compensate for the increased ISO, this surely helps against blooming. No question about that.

But the ISO increase by itself does nothing to help with blooming (other than smearing the distinct disc outlines with noise and better gradations), light exposure of the sensor doesn't change. Most people seem to automatically connect an ISO increase with an exposure decrease, but that's not a given (and a good reason to do tests in M-anual mode for better control of what's happening).
 
As Silkypix was the 'Raw translator' of choice by Fuji, I cannot see how it is designated "third party". Purely by my definition and as it's the supplied item it is the primary converter although made by a company other than Fuji. It is after all supplied by Fuji.

Dave. (UK)
I wouldn't make this assumption, I was the UK distributor for Silkypix / ISL.

Most software companies approach the camera companies and offer a bundled solution in order to generate sales and spin offs.

The software company then extends product licensing to the camera manufacturer (nominal unit cost) and they rebadge it.
I'm guessing ISL independently (not Fuji) did the code work for RAF files.

Although Fuji may have released limited RAF specs to ISL. However, ISL then have to interpret these in the context of their RAW engine, which may have limitations for this type of application.
 
Tell that to Adobe, Phase One and ISL. :P My guess is that they don't want to invest the manpower necessary to cover the whole EXR + DR range, while all other cams work with simple Bayer patterns.
Have to agree on this!

I've already posed this Q to Phase One, as the X10=S3 upscaling rhetoric don't wash.
 
As Silkypix was the 'Raw translator' of choice by Fuji, I cannot see how it is designated "third party". Purely by my definition and as it's the supplied item it is the primary converter although made by a company other than Fuji. It is after all supplied by Fuji.

Dave. (UK)
I wouldn't make this assumption, I was the UK distributor for Silkypix / ISL.

Most software companies approach the camera companies and offer a bundled solution in order to generate sales and spin offs.

The software company then extends product licensing to the camera manufacturer (nominal unit cost) and they rebadge it.
I'm guessing ISL independently (not Fuji) did the code work for RAF files.

Although Fuji may have released limited RAF specs to ISL. However, ISL then have to interpret these in the context of their RAW engine, which may have limitations for this type of application.
I think that my assumption is a logical one although maybe not actuality. Presuming what you say may be correct and there is no 'in house' Fuji converter, it in effect means that Fuji have sold a camera with a raw output (a dealbreaker to some) with no means of properly resolving the files! Somewhere along the line there must be some intent to deceive by Fuji. The same thing still applies if a raw converter that re-assembles the sensor output correctly is too difficult or impossible to manufacture.

Obviously a case of what you see is NOT what you get :)

Dave. (UK)
 
Fujifilm would probably argue that the camera does all the neccessary RAW processing / conversion, so providing RAF files to the end user is icing.
Yes John - you're probably correct. There was me thinking there was still a touch of virtue left in this world. :(

Dave. (UK)
I suppose that there is always a chance of instant fame for a code writer who could successfully write a programme to properly convert EXR sensor output. No......??

Dave. (UK)
 
After further evaluation I have to give the current RAW processor crown to Lightroom 4 (Beta) and correct the statement that all RAW converters in my test lose saturation in dark images.

It turned out that the loss of saturation in low light pictures is a result of color noise filtering !

In Lightroom this is not a problem. The default setting chosen by LR is far too high, but one move of a slider fixes it easily, so you can call that user error on my first tries.

Silkypix 4 (Raw Converter EX) and SP 5 don't seem to allow manual control over color noise filtering, but make the whole noise filtering business a part of the sharpness setting. Even worse SP seems to decide somewhat arbitrarily how much color noise filtering to apply. SP4/EX vs. 5 produce different results with the same images.
  • On one image SP4/EX apply so much color noise filtering as if you would pull the slider in LR to max . This leads to serious color bleed (red wall color bleeds heavily into black speaker), while SP5 does not suffer from this.
  • On another image SP4/EX preserve more color/saturation than SP5, behaving as if SP4 uses less color noise filtering vs. SP5 (same settings).
SP5 is still a Beta and it generally has some quirks with its sharpening settings (the first image I talk about above is smoothed to death using the same settings as SP4). But both suffer from their lack of real control (at least I couldn't find any).

Lightroom 4 also is the one software that can pull more highlight detail out of clipped areas in ISO DR 400 images than any other contender (including LR 3), especially without having to underexpose the whole image. So that "Highlight" slider is really useful! It still seems to push some reds a bit too far into orange compared to JPGs, but pulling the Orange slider down to about -10 fixes that, too.

I will give LR4 another run with the 12 MP L DR 100 image to see if I can get out more detail without over-sharpening. Too bad that EXP DR controls are lacking and colors of the longer exposed clipping areas are off. Hopefully LR4 Beta can be used until DXO come out with their X10 support (is there any way to contact that company for pre_sales questions?).
 
Two things to take note off:

Lightroom's color noise filter allows to further reduce colored moire patterns that the JPGs leave in. Curiously I got more of that moire (on black colored canvas) from M size than from L size shots. Differences ain't big though.

Unfortunately Lightroom's luminance noise filter is less effective than the in-camera one, it blurs more detail away (regardless of the the Detail slider value).
 
I wrote a mail to DxO today, asking how their promised future support of the X10 is going to be implemented in detail, especially concerning EXR exposure time split DR modes that all other RAW converters fail atm. Once I get an answer I will let you know.

One more disadvantage I noticed with Lightroom is its sharpening filter. It's unsharp mask range is quite limited, in return you get the "Detail" slider. Unfortunately that slider mostly pulls out the EXR labyrinth like noise pattern and turns the image into an noisy mess.
 
Timur - Now as you have been exploring (in depth) the decoding and file handing deficiencies of several raw conversion programmes a thought strikes me. The inabillity of said programmes to fully exploit the EXR files may account for the difficulty of others to properly match the jpeg quality with their attempts at raw conversion. I myself have achieved the rescue of some of my raw shots from over exposure but even after this the overall quality of the final result did not exceed the correctly exposed jpeg output.

Yes it's possible to put in custom settings for more exact noise reduction and generally (with more control) fiddle with the colours and sharpening but this is more for personal liking and doesn't generally 'give' the picture the extra info' it would benefit from to improve quality.

Before the pedants and measurebators move in, these are my own personal thoughts on the subject. All comments and useful views however are welcome.

Dave. (UK)
 
Some differences surely account to the difference in how the sensor halves are worked with. But judging on the not soo big differences of the L size images I would attribute a lot of the differences on the filters!

I'd have to look at Silkypix again, but for Lightroom I can tell you that:
  • The unsharp (sharpening) filter has a rather limited range. Instead you are supposed to use the "Detail" slider, but this slider is nearly useless for X10 (EXR) files. That is because it pulls out the specific EXR labyrinth like noise and turns the image into a noisy mess pretty easily.
  • The luminance noise filter blurs more detail away than in-camera noise-filtering when set to keep the same level of detail. Using the "Detail" slider can help, but comes at the cost of sharpening outlines and still keeps areas blurry. (DxO works a bit different there.)
  • Red saturation generally works different for ooc JPGs and RAWs and using ooc JPG M size additionally affects white-balance in that it protects "warmer" WB that would get lost with higher ISO. No idea how RAW software handles that yet.
 
http://www.visualbakery.com/RawTherapee/Downloads.aspx

It'll be interesting to see what highlights details RT can extract from the X10 -- usually it's one of the best programs in this respect.
Took a very quick look: Highlight extraction seems to be the same as in Lightroom, but less comfortable (the new "Highlight" slider in LR4 is very useful).

Unfortunately RT does not support EXR DR files with different exposure times for half the sensor. It only seems to use the data from the longer exposed half, throwing away half the information.

Unsurprisingly automated lens correction is not performed for the X10, but you can do it manually.
 
I wrote a mail to DxO today, asking how their promised future support of the X10 is going to be implemented in detail, especially concerning EXR exposure time split DR modes that all other RAW converters fail atm. Once I get an answer I will let you know.
Can you point me to your posts where you explain and demonstrate this theory of yours?

--
Cheers ;-)

Trevor G

Silkypix tutorials at: http://photo.computerwyse.com
 
Second post in this very thread holds the explanation. No demonstration images up.
 

Keyboard shortcuts

Back
Top