Compressed RAFs

Started May 4, 2016 | Discussions
Iliah Borg Forum Pro • Posts: 25,641
Compressed RAFs
5

We are preparing an open source release of decompressor as a part of LibRaw. That will allow raw converters, including open source ones like RawTherapee, Darktable, UFRaw to decode FujiFilm compressed raw files.

To thoroughly test the decoder we incorporated it into our FastRawViewer. Please do try it and let us know if there are any files that cause problems.

Up-to-date manual

You can always contact us at support@fastrawviewer.com

-- hide signature --
uMad Regular Member • Posts: 477
Re: Compressed RAFs

great work - thank you!

did you have to reverse engineer it or did you have had help by Fuji?

 uMad's gear list:uMad's gear list
Fujifilm X-T2 Fujifilm GFX 50S Fujifilm X-H1 Fujifilm GF 110mm F2 Fujifilm GF 23mm F4
OP Iliah Borg Forum Pro • Posts: 25,641
Re: Compressed RAFs

uMad wrote:

great work - thank you!

did you have to reverse engineer it or did you have had help by Fuji?

No help from Fuji, unfortunately.

-- hide signature --
Alexey.Danilchenko Senior Member • Posts: 1,242
Re: Compressed RAFs
3

uMad wrote:

great work - thank you!

did you have to reverse engineer it or did you have had help by Fuji?

If we'd used Fuji SDK (their SDK used by Silkypix, Iridient and Adobe) then conversion time would be atrocious. Our code works *much* faster and decoding can be done concurrently to speed it up.

uMad Regular Member • Posts: 477
Re: Compressed RAFs

that's what I thought. Too bad.

But thanks for doing the hard work!

 uMad's gear list:uMad's gear list
Fujifilm X-T2 Fujifilm GFX 50S Fujifilm X-H1 Fujifilm GF 110mm F2 Fujifilm GF 23mm F4
uMad Regular Member • Posts: 477
Re: Compressed RAFs

AlexeyD wrote:

uMad wrote:

great work - thank you!

did you have to reverse engineer it or did you have had help by Fuji?

If we'd used Fuji SDK (their SDK used by Silkypix, Iridient and Adobe) then conversion time would be atrocious.

Interesting. Nice work!

 uMad's gear list:uMad's gear list
Fujifilm X-T2 Fujifilm GFX 50S Fujifilm X-H1 Fujifilm GF 110mm F2 Fujifilm GF 23mm F4
Sal Baker Forum Pro • Posts: 11,363
Re: Compressed RAFs

AlexeyD wrote:

uMad wrote:

great work - thank you!

did you have to reverse engineer it or did you have had help by Fuji?

If we'd used Fuji SDK (their SDK used by Silkypix, Iridient and Adobe) then conversion time would be atrocious. Our code works *much* faster and decoding can be done concurrently to speed it up.

How long does it take for Iridient to convert a compressed RAF on a moderately fast computer?  Thanks.

Sal

 Sal Baker's gear list:Sal Baker's gear list
Ricoh GR Fujifilm X-E2 Fujifilm X-T100 Fujifilm XF 14mm F2.8 R Fujifilm XF 18-55mm F2.8-4 R LM OIS +3 more
Alexey.Danilchenko Senior Member • Posts: 1,242
Re: Compressed RAFs

Sal Baker wrote:

How long does it take for Iridient to convert a compressed RAF on a moderately fast computer? Thanks.

The decoding time for compressed RAFs depends not just on Iridient code but on the unpacking routines used by Fuji SDK (I presume it was SDK since it comes as DLL and is the same in Iridient and Silkypix with the same code found embedded in DNG converter).

I therefore can give you the measurements I have done on iMac 27" (March 2016 one) with i7 CPU and 32 GB RAM - decoding (just decoding) of single file takes about 8 seconds (if I recall this correctly from my initial experiments). The FRV (Fast Raw Viewer) that uses our code does decode single file in subsecond time frame.

sunhorse Senior Member • Posts: 1,718
Re: Compressed RAFs

I am running OS X 10.11.5 Beta (15F31a). I am seeing some artifacts on highlighted edges.

I can provide a Dropbox or Google Drive link to the original RAF file if you would like to have the original. Please let me know.

-- hide signature --

Ramesh

 sunhorse's gear list:sunhorse's gear list
Fujifilm XF10 Fujifilm X-Pro2 Fujifilm XF 18mm F2 R Fujifilm XF 60mm F2.4 R Macro Fujifilm XF 18-55mm F2.8-4 R LM OIS +4 more
OP Iliah Borg Forum Pro • Posts: 25,641
Re: Compressed RAFs
1

sunhorse wrote:

I am running OS X 10.11.5 Beta (15F31a). I am seeing some artifacts on highlighted edges.

I can provide a Dropbox or Google Drive link to the original RAF file if you would like to have the original. Please let me know.

That's because FastRawViewer is in fast demosaicking mode. You can improve the rendition a little, Preferences -> Performance -> uncheck "Use high speed previews for Fujifilm X-Trans".

-- hide signature --
bowportes Veteran Member • Posts: 3,544
Re: Compressed RAFs

What would really be cool would be if your rawviewer could pass uncompreesed raw data on to an external editor such as SP so we don't have to wait for them to deal with the compressed raw

 bowportes's gear list:bowportes's gear list
Panasonic Lumix DMC-G5 Fujifilm X-M1 Fujifilm X-T1 Fujifilm X-Pro2 Panasonic Lumix G 20mm F1.7 ASPH +13 more
OP Iliah Borg Forum Pro • Posts: 25,641
Re: Compressed RAFs

bowportes wrote:

What would really be cool would be if your rawviewer could pass uncompreesed raw data on to an external editor such as SP so we don't have to wait for them to deal with the compressed raw

The hope is that they will look at our open source decoder and use it instead, gaining speed. With FastRawViewer used for un-packing the workflow and backup strategy become convoluted.

-- hide signature --
Shawn67 Senior Member • Posts: 2,100
Re: Compressed RAFs
1

bowportes wrote:

What would really be cool would be if your rawviewer could pass uncompreesed raw data on to an external editor such as SP so we don't have to wait for them to deal with the compressed raw

You can get around this with Adobe DNG Converter. Use 9.4x and be sure to set it to Camera Raw 7.1 and later. If you do that it will convert to DNG but not demosaic the file. Setting it to earlier versions of Camera RAW will result in Adobe demosaicing the X-Trans data.

Kind of a pain though so it could just be easier to shoot uncompressed until support comes to SP then switch over to compressed RAW files.

Shawn

OP Iliah Borg Forum Pro • Posts: 25,641
Re: Compressed RAFs

Shawn67 wrote:

bowportes wrote:

What would really be cool would be if your rawviewer could pass uncompreesed raw data on to an external editor such as SP so we don't have to wait for them to deal with the compressed raw

You can get around this with Adobe DNG Converter. Use 9.4x and be sure to set it to Camera Raw 7.1 and later. If you do that it will convert to DNG but not demosaic the file. Setting it to earlier versions of Camera RAW will result in Adobe demosaicing the X-Trans data.

Kind of a pain though so it could just be easier to shoot uncompressed until support comes to SP then switch over to compressed RAW files.

Shawn

SP = SilkiPix? If so, they use Fujifilm SDK to support compressed RAFs. The only problem with that decoding is that it is a bit slow.

-- hide signature --
Alexey.Danilchenko Senior Member • Posts: 1,242
Re: Compressed RAFs

SP = SilkiPix? If so, they use Fujifilm SDK to support compressed RAFs. The only problem with that decoding is that it is a bit slow.

More than a bit, I just tried it again on Windows PC on the same compressed RAF - 11 seconds just calling the Fuji SDK (decoding part) vs 3.5 seconds in our LibRaw version (non concurrent version for the direct comparison - with concurrency ours is even faster)

Shawn67 Senior Member • Posts: 2,100
Re: Compressed RAFs

AlexeyD wrote

The decoding time for compressed RAFs depends not just on Iridient code but on the unpacking routines used by Fuji SDK (I presume it was SDK since it comes as DLL and is the same in Iridient and Silkypix with the same code found embedded in DNG converter).

I therefore can give you the measurements I have done on iMac 27" (March 2016 one) with i7 CPU and 32 GB RAM - decoding (just decoding) of single file takes about 8 seconds (if I recall this correctly from my initial experiments). The FRV (Fast Raw Viewer) that uses our code does decode single file in subsecond time frame.

I am getting similar times on a Mid 2015 MacBook Pro 2.5Ghz i7 and 16 gB of memory with a SSD. Iridient Developer opens the file in 7-8 seconds, FastRawViewer is about a second.

Compressed support coming to RPP64?

Thanks,

Shawn

Alexey.Danilchenko Senior Member • Posts: 1,242
Re: Compressed RAFs
Compressed support coming to RPP64?

From what Iliah told me - yes

OP Iliah Borg Forum Pro • Posts: 25,641
Re: Compressed RAFs

AlexeyD wrote:

SP = SilkiPix? If so, they use Fujifilm SDK to support compressed RAFs. The only problem with that decoding is that it is a bit slow.

More than a bit, I just tried it again on Windows PC on the same compressed RAF - 11 seconds just calling the Fuji SDK (decoding part) vs 3.5 seconds in our LibRaw version (non concurrent version for the direct comparison - with concurrency ours is even faster)

I'm trying hard not to be too hard on Fujifilm policies :))

-- hide signature --
bowportes Veteran Member • Posts: 3,544
Re: Compressed RAFs

Iliah Borg wrote:

AlexeyD wrote:

SP = SilkiPix? If so, they use Fujifilm SDK to support compressed RAFs. The only problem with that decoding is that it is a bit slow.

More than a bit, I just tried it again on Windows PC on the same compressed RAF - 11 seconds just calling the Fuji SDK (decoding part) vs 3.5 seconds in our LibRaw version (non concurrent version for the direct comparison - with concurrency ours is even faster)

I'm trying hard not to be too hard on Fujifilm policies :))

It would be nice if you could display the Fuji-specific EXIF information -- Color, Shadow, Highlights, DR, NR, Sharpening, and Film Simulation -- along with the more generic stuff.

 bowportes's gear list:bowportes's gear list
Panasonic Lumix DMC-G5 Fujifilm X-M1 Fujifilm X-T1 Fujifilm X-Pro2 Panasonic Lumix G 20mm F1.7 ASPH +13 more
bowportes Veteran Member • Posts: 3,544
Re: Compressed RAFs

You also appear not to be applying the RAF file's distortion correction information in your fastrawviewer display.

 bowportes's gear list:bowportes's gear list
Panasonic Lumix DMC-G5 Fujifilm X-M1 Fujifilm X-T1 Fujifilm X-Pro2 Panasonic Lumix G 20mm F1.7 ASPH +13 more
Keyboard shortcuts:
FForum MMy threads