X3 Developer - calling all software minds..

What's wrong with using your X3F tools and simply export the photographs to TIFFs and import that the image editor of ones choice?

Do one really need anything more?
A way to convert the X3 RAW layers to RGB.
Is that what your tool does? One can save the images as either JPG or TIFF which are both RGB.
No

My tool mainly do two things.
  1. It extracts the embedded JPEG.
  2. It reads the encrypted/compressed RAW X3 data and saves it as layers in a TIFF file.
The RAW X3 data is not RGB. It has the same structure as an RGB image, so it can be saved in a TIFF.

Therefore, a step that converts this RAW data to RGB is needed.

For Quattro, there is an extra complication. The layers are not the same resolution. So .... it cannot be saved in a TFF as it is. Therefore, the tool expands the lower layers from 5 MP to 20 MP before saving to the TIFF.
Then it imperative that we fist and foremost back engineer RAW to RGB and create a library to handle that.
 
I have a pretty fair amount of experience with GPUImage, having used it in two commercial apps for iOS. It has a 4k x 4k size limit for its textures ingrained in the code, a problem that Brad Larson has acknowledged but probably won't fix until an Apple camera has a resolution that requires increasing that limit.

I tried to do some stress testing of the apps with Merrill files, and it didn't work so well for that very reason.

Just in case this turns into a serious project, I think whatever software results from this thread should have the ability to utilize the GPU as well as the CPU, when processing Foveon raw files, so it can be as fast as possible. Apparently Capture One 7 Pro has this capability now, and I dare say it could possibly work very fast on a good gaming machine with a $500 video card in it. Kendall posted a very interesting link in another thread, which lead me to something else that is very interesting, which might be useful for the implementation of such a feature:

https://github.com/BradLarson/GPUImage
 
Iridient is mac only which excludes 85% or whatever the current market share is of PC users.
The "market share" of PC systems sold includes company computers that sit on desks and do something simple.

Much more telling is that Apple has something like 80% of the high-end laptop market share now...

Go to a photography conference and the estimate for Mac use by photographers is way north of 50%, not anything like 25%.
Not a great plan. I quite like the idea of helping sigma develop spp jor even a simple batch converter but I've heard Japanese companies aren't keen on pushing things outside the family so they are unlikely to play ball?
Don't think that's the issue.
 
My tool mainly do two things.
  1. It extracts the embedded JPEG.
  2. It reads the encrypted/compressed RAW X3 data and saves it as layers in a TIFF file.
The RAW X3 data is not RGB. It has the same structure as an RGB image, so it can be saved in a TIFF.

Therefore, a step that converts this RAW data to RGB is needed.

For Quattro, there is an extra complication. The layers are not the same resolution. So .... it cannot be saved in a TFF as it is. Therefore, the tool expands the lower layers from 5 MP to 20 MP before saving to the TIFF.
 
Last edited:
Why not simply work on implementing X3F support into the Darktable software?
Yes, that is an idea.

Have not checked the actual status of DarkTable.
I wouldn't bother because I went to their site and, unlike other sites, nowhere is there a list of supported formats, unless I missed it. Apparently you have download their stuff, then try it with a file and, if nothing happens, the format is not supported.

I'm trying Qimage which does open X3Fs (at least SD9 thru SD14 and DP1) but it's main purpose seems to be for batch printing and I don't print. I find the user interface a little odd but it appears to work and has a good few tools.

I'm still trying to gauge whether this thread wants a bare-bones converter (raw to TIFF) or a full-blown equivalent of PhotoShop?
 
My tool mainly do two things.
  1. It extracts the embedded JPEG.
  2. It reads the encrypted/compressed RAW X3 data and saves it as layers in a TIFF file.
The RAW X3 data is not RGB. It has the same structure as an RGB image, so it can be saved in a TIFF.

Therefore, a step that converts this RAW data to RGB is needed.

For Quattro, there is an extra complication. The layers are not the same resolution. So .... it cannot be saved in a TFF as it is. Therefore, the tool expands the lower layers from 5 MP to 20 MP before saving to the TIFF.
 
My tool mainly do two things.
  1. It extracts the embedded JPEG.
  2. It reads the encrypted/compressed RAW X3 data and saves it as layers in a TIFF file.
The RAW X3 data is not RGB. It has the same structure as an RGB image, so it can be saved in a TIFF.

Therefore, a step that converts this RAW data to RGB is needed.

For Quattro, there is an extra complication. The layers are not the same resolution. So .... it cannot be saved in a TFF as it is. Therefore, the tool expands the lower layers from 5 MP to 20 MP before saving to the TIFF.
Hey Roland-

I briefly mentioned this before (I think in this thread) but I think it would be cool of there was an X3F converter that was based on the (presumed) mathematics of the Sigma preset white balance values, which I believe are a 3x3 matrix (too/middle/bottom x r/g/b).
Several matrices, actually, Mark. The most significant perhaps being the one that transforms camera raw values into CIE XYZ co-ordinates.

As a bonus for pre-Merrill users and maybe pre-AFE users I offer a bare-bones toy where a cam to RGB matrix is actually adjustable!


Unzipping gives three folders: Linux, OSX, Windows versions.

Couldn't find it on CEZEO's site so it's up on mine for a limited time.
If I could input them into something like your x3f tools

x3f_extract -tiff tr=-0.1 -mr=0.4 -br=0.6 -tg=0.4 -mg=0.6 -bg=-0.2 -tb=0.8 -mb=0.2 -bb=-0.1 -foo.x3f

E.g. the "red" layer becomes

Top -0.1
Middle 0.4
Bottom 0.6

Mark
Nice proposal.

I am right now considering what to do - so all suggestions are welcome.
For my camera models (and according to ArvoJ) the cam-to-XYZ matrix is different for each sensor so as to match the variable space per sensor to the WB matrices, etc. That is each individual sensor, not sensor or camera model. So, if that is true for all Sigma camera models, would you be using that matrix as part of your RGB extractor?

The CEZEO reader apparently does not and, with default matrix co-efficients set, can be dark (no auto-brightening) and is a bit heavy on purples (i.e. red and blue content). It does do the same or similar to your raw extractor if you alter the CEZEO matrix to a 1,1,1 diagonal.
 
  • Like
Reactions: mmr
I have a pretty fair amount of experience with GPUImage, having used it in two commercial apps for iOS. It has a 4k x 4k size limit for its textures ingrained in the code, a problem that Brad Larson has acknowledged but probably won't fix until an Apple camera has a resolution that requires increasing that limit.

I tried to do some stress testing of the apps with Merrill files, and it didn't work so well for that very reason.
Just in case this turns into a serious project, I think whatever software results from this thread should have the ability to utilize the GPU as well as the CPU, when processing Foveon raw files, so it can be as fast as possible. Apparently Capture One 7 Pro has this capability now, and I dare say it could possibly work very fast on a good gaming machine with a $500 video card in it. Kendall posted a very interesting link in another thread, which lead me to something else that is very interesting, which might be useful for the implementation of such a feature:

https://github.com/BradLarson/GPUImage
Why does it have a 4K x 4K limit? Is it a speed issue? Can't that be doubled easily? I'm thinking that if it's a program designed for manipulating images it needs to be modified ASAP, because even some of the cameras in cell phones out there now make photos that are more than 3,780 pixels on the longest side - some are more than 5,000 pixels on the longest side. Doubling the dimensions would make the program work for every popular digital camera for years to come (even for Quattro files exported from SPP at S-HI size). If it's a speed issue, I would think it might make sense to give the program a "hi-res" mode, which can be switched on and off, so people CAN use larger images with it, but more slowly. Have you tried to play with the code, to increase the size of images it can handle?
 
Why not simply work on implementing X3F support into the Darktable software?
Yes, that is an idea.

Have not checked the actual status of DarkTable.
I wouldn't bother because I went to their site and, unlike other sites, nowhere is there a list of supported formats, unless I missed it. Apparently you have download their stuff, then try it with a file and, if nothing happens, the format is not supported.

I'm trying Qimage which does open X3Fs (at least SD9 thru SD14 and DP1) but it's main purpose seems to be for batch printing and I don't print. I find the user interface a little odd but it appears to work and has a good few tools.

I'm still trying to gauge whether this thread wants a bare-bones converter (raw to TIFF) or a full-blown equivalent of PhotoShop?
 
I have a pretty fair amount of experience with GPUImage, having used it in two commercial apps for iOS. It has a 4k x 4k size limit for its textures ingrained in the code, a problem that Brad Larson has acknowledged but probably won't fix until an Apple camera has a resolution that requires increasing that limit.

I tried to do some stress testing of the apps with Merrill files, and it didn't work so well for that very reason.
Just in case this turns into a serious project, I think whatever software results from this thread should have the ability to utilize the GPU as well as the CPU, when processing Foveon raw files, so it can be as fast as possible. Apparently Capture One 7 Pro has this capability now, and I dare say it could possibly work very fast on a good gaming machine with a $500 video card in it. Kendall posted a very interesting link in another thread, which lead me to something else that is very interesting, which might be useful for the implementation of such a feature:

https://github.com/BradLarson/GPUImage
Why does it have a 4K x 4K limit? Is it a speed issue? Can't that be doubled easily? I'm thinking that if it's a program designed for manipulating images it needs to be modified ASAP, because even some of the cameras in cell phones out there now make photos that are more than 3,780 pixels on the longest side - some are more than 5,000 pixels on the longest side. Doubling the dimensions would make the program work for every popular digital camera for years to come (even for Quattro files exported from SPP at S-HI size). If it's a speed issue, I would think it might make sense to give the program a "hi-res" mode, which can be switched on and off, so people CAN use larger images with it, but more slowly. Have you tried to play with the code, to increase the size of images it can handle?
It's a GL version issue. See here:



Basically, the GPU is intended to draw things to the screen. The fact that it can be used to process larger images (or other things, like physics simulation) is a happy accident, but not the main purpose of the subsystem. Brad's library is mainly about capturing video and stills from the iPhone and mac cameras, and then manipulating those stills. As Apple has yet to release a camera larger than 4k x 4k, there's no pressing need to increase the texture size, or to implement tiling (ie, stitching together smaller textures and having some code to handle the border cases).
 
I have a pretty fair amount of experience with GPUImage, having used it in two commercial apps for iOS. It has a 4k x 4k size limit for its textures ingrained in the code, a problem that Brad Larson has acknowledged but probably won't fix until an Apple camera has a resolution that requires increasing that limit.

I tried to do some stress testing of the apps with Merrill files, and it didn't work so well for that very reason.
Just in case this turns into a serious project, I think whatever software results from this thread should have the ability to utilize the GPU as well as the CPU, when processing Foveon raw files, so it can be as fast as possible. Apparently Capture One 7 Pro has this capability now, and I dare say it could possibly work very fast on a good gaming machine with a $500 video card in it. Kendall posted a very interesting link in another thread, which lead me to something else that is very interesting, which might be useful for the implementation of such a feature:

https://github.com/BradLarson/GPUImage
Why does it have a 4K x 4K limit? Is it a speed issue? Can't that be doubled easily? I'm thinking that if it's a program designed for manipulating images it needs to be modified ASAP, because even some of the cameras in cell phones out there now make photos that are more than 3,780 pixels on the longest side - some are more than 5,000 pixels on the longest side. Doubling the dimensions would make the program work for every popular digital camera for years to come (even for Quattro files exported from SPP at S-HI size). If it's a speed issue, I would think it might make sense to give the program a "hi-res" mode, which can be switched on and off, so people CAN use larger images with it, but more slowly. Have you tried to play with the code, to increase the size of images it can handle?
It's a GL version issue. See here:

http://stackoverflow.com/questions/5506426/rendering-large-textures-on-ios-opengl

http://stackoverflow.com/questions/17186655/ios-retina-image-size-limit

Basically, the GPU is intended to draw things to the screen. The fact that it can be used to process larger images (or other things, like physics simulation) is a happy accident, but not the main purpose of the subsystem. Brad's library is mainly about capturing video and stills from the iPhone and mac cameras, and then manipulating those stills. As Apple has yet to release a camera larger than 4k x 4k, there's no pressing need to increase the texture size, or to implement tiling (ie, stitching together smaller textures and having some code to handle the border cases).
In CoreImage with iOS8/Yosemite Apple is now doing automatic tiling under the hood. So writing a Core Image filter to manipulate things is no longer limited to 4k x 4k, and even on mobile devices applications can now work with any size image.
 
I'm still trying to gauge whether this thread wants a bare-bones converter (raw to TIFF) or a full-blown equivalent of PhotoShop?
Ted I think we all probably want a combination - a minimal, plug-in style bare-bones converter, which will make Photoshop and GIMP compatible with SD15, Merrill and Quattro raw files, AND a Lighroom/SPP replacement, which has more capabilities for reviewing, cataloging, adjusting, cropping, and converting raw files from every Sigma digital camera (and quite possibly every other raw file format that's supported by LibRaw or DCraw).
Well, RawDigger uses LibRaw and supports all Sigma cameras up to and including Merrill, so that's the bare-bones bit right there, eh? But Iliah and Alex seem to have missed this thread - or maybe there's a Great Silence happening there . .
 
CoreImage != GPUImage. He may decide to start doing that under the hood, but last I checked, it was all pretty much straight up OpenGL ES.

Can you point me to a link about automatic tiling? Because I've always had to be very careful about edge conditions when doing tiling if doing any kind of nontrivial filtering involving more than just the current pixel, I'd be cautious about how they're doing this.
 
CoreImage != GPUImage. He may decide to start doing that under the hood, but last I checked, it was all pretty much straight up OpenGL ES.
I know, I'm just saying that at least a foundation exists that does do tiling that something could be written on top of.
Can you point me to a link about automatic tiling? Because I've always had to be very careful about edge conditions when doing tiling if doing any kind of nontrivial filtering involving more than just the current pixel, I'd be cautious about how they're doing this.
I don't think they've produced anything with much technical detail yet, all I know is a summary of what is possible from watching a WWDC 2014 video on "what is new in Core Image" (which also talks about custom CoreImage filters for the RAW image pipeline).

Apple in general though has a ton of image processing expertise, so I'm pretty sure they are doing a careful job of this.
 
So Kendall, how much do you know about GPU image processing accelleration? With a raw image processor capable of using the GPU, would it make sense to get a slower computer for $200 less and add a second identical video card for $150, saving money and increasing image processing power, or does having multiple video cards not matter, when using GPU-based image processing?

Ted, from what you said about LibRaw, I'm inclined to believe that "engine" is probably superior to DCraw becuase of this. What do YOU think?
 
So Kendall, how much do you know about GPU image processing accelleration? With a raw image processor capable of using the GPU, would it make sense to get a slower computer for $200 less and add a second identical video card for $150, saving money and increasing image processing power, or does having multiple video cards not matter, when using GPU-based image processing?
It depends on the application, for some applications another video card does add more processing power. If it's just doing simpler calculations on the GPU the system would just use the fastest one.

Neither would help SPP at the moment though. I think in general you are always well off getting near to the top (but not quite top) of the line CPU, and a fairly powerful GPU if possible to help future-proof your purchase as much as you can.
Ted, from what you said about LibRaw, I'm inclined to believe that "engine" is probably superior to DCraw becuase of this. What do YOU think?
It seems like it would be a better choice for a RAW conversion library if you were building a new app.
 
Search for "gpu jpeg compression" on google.

It seems to be a rather popular topic.
 
I'm still trying to gauge whether this thread wants a bare-bones converter (raw to TIFF) or a full-blown equivalent of PhotoShop?
Ted I think we all probably want a combination - a minimal, plug-in style bare-bones converter, which will make Photoshop and GIMP compatible with SD15, Merrill and Quattro raw files, AND a Lighroom/SPP replacement, which has more capabilities for reviewing, cataloging, adjusting, cropping, and converting raw files from every Sigma digital camera (and quite possibly every other raw file format that's supported by LibRaw or DCraw).
Well, RawDigger uses LibRaw and supports all Sigma cameras up to and including Merrill, so that's the bare-bones bit right there, eh? But Iliah and Alex seem to have missed this thread - or maybe there's a Great Silence happening there . .
 

Keyboard shortcuts

Back
Top