Uncompressed RAW and DNG converter

[randomguy]

I suggest compressing them with a file archiver when archiving the files. 7-zip reduced the file size of my uncompressed files from today 57% which I expect is significantly better than you'll ever get with some computationally cheap on camera lossless compression (and those lossless compressed files are very hard to compress further). That the files are big while on the memory card or while on my HD doesn't bother me, just means I'll be archiving more often.

 

[Esstee]

I haven't done the update yet(waiting 'til evening), but wanted to ask what the ave. size of the uncompressed files are?

 

[juvx]

I haven't done the update yet(waiting 'til evening), but wanted to ask what the ave. size of the uncompressed files are?

80-85 MB range.

 

[exapixel]

I haven't done the update yet(waiting 'til evening), but wanted to ask what the ave. size of the uncompressed files are?

It's two bytes per pixel, exactly, plus a megabyte or so of JPEG preview and metadata.

 

[Cheyne]

So, how big are these new uncompressed RAW files?

The regular RAW was consistently around 42MB for me, wondering how much it's grown now.

About 83MiB. The per-pixel data doubles in size, from one byte per pixel to two bytes per pixel. The fixed overhead and small JPEG preview image are of course unchanged in size.

Yes, it's big. Yes, it's lame.

 

[Iliah Borg]

ARW:

DNG:

That is, the optical black is lost in DNG, and part of the data is replaced with garbage (that's the right part of the uncompressed raw from A7S2)



--

About LibRaw | LibRaw

www.libraw.org

 

[jackalopemaui]

ARW:



DNG:



That is, the optical black is lost in DNG, and part of the data is replaced with garbage (that's the right part of the uncompressed raw from A7S2)

 

[juvx]

So, how big are these new uncompressed RAW files?

The regular RAW was consistently around 42MB for me, wondering how much it's grown now.

About 83MiB. The per-pixel data doubles in size, from one byte per pixel to two bytes per pixel. The fixed overhead and small JPEG preview image are of course unchanged in size.

Yes, it's big. Yes, it's lame.

 

[Iliah Borg]

ARW:

DNG:

That is, the optical black is lost in DNG, and part of the data is replaced with garbage (that's the right part of the uncompressed raw from A7S2)

Interesting... So Adobe's lossless DNG isn't really lossless??

When you convert, there is always some risk that the data is lost, especially with the first converter versions. Theoretically it should be lossless, less any shortcuts and programming errors.

 

[juvx]

ARW:

DNG:

That is, the optical black is lost in DNG, and part of the data is replaced with garbage (that's the right part of the uncompressed raw from A7S2)

 

[Iliah Borg]

In reality there is no such thing as lossless and Compressed in one sentence.

Do you think LZW or ZIP or RAR or bzip are lossy?

 

[juvx]

In reality there is no such thing as lossless and Compressed in one sentence.

Do you think LZW or ZIP or RAR or bzip are lossy?

 

[exapixel]

In reality there is no such thing as lossless and Compressed in one sentence.

Do you think LZW or ZIP or RAR or bzip are lossy?

 

[Esstee]

Eh .. I didn't have a problem with the previous RAW, think ill just keep using that.

I don't see how that's a problem

I would be curious to see how many working professionals are coming across limitations of the compressed RAW format.

In my own case, this would be the wrong term to approach with.
However, I never liked lossy compression in RAW, never did and never will as it comes at the cost of the fidelity of the file. Which isn't to say that this would ever affect the casual shooter, but more along the lines of those seeking to use of as much of the sensor data as possible. Such as with landscape shots(for example) where the only recourse for dynamic range is with multiple exposures. In contrast to that of using a single file with shadow recovery(for example).

Beyond this, there are other instances where uncompressed RAW file performances may become a factor, such as in low light(night street photography). Which would definitely benefit from uncompressed data also.

Having said all that, it's worth pointing out that identifying such instances would have no impact on the existing satisfaction of others who would a) have interest in such shooting and processing method or b) be satisfied with compressed RAW files. - and so I think it's important to distinguish this also.

 

[Esstee]

I haven't done the update yet(waiting 'til evening), but wanted to ask what the ave. size of the uncompressed files are?

80-85 MB range.

Awesome.

And definitely not an issue insofar as file size imo. ie, 128GB SDxC cards, and 4TB storage drives would handle this just fine imo. Which won't likely be for everyone as these will no doubt cater to specific shooting styles.

 

[mgrum]

Interesting... So Adobe's lossless DNG isn't really lossless??

The format is (when in lossless mode) but Adobe's own converter isn't (third party converters are possible). An important distinction to make...

 

[mgrum]

I meant to edit to explain more / better haha but it wont let me.

I meant specifically for RAW files. There will always be a little tiny bit of degradation due to errors / programming, etc.

Not necessarily. True 100% lossless compression is entirely possible.

Speaking of which iv always wondered why a camera manufacturer doesn't impediment a straight up RAR format compression (or other) to the raw file in camera.. Cameras are fast enough to compress on the fly now. It really shouldn't be an issue.

Most use lossless JPEG. Other algorithms are possible, image specific algorithms ought to outperform RAR, which isn't particularly sophisticated (it's an LZ77 derivative - the 77 refers to 1977, i.e. it's older than Star Wars).

 

[Rod Laird]

....
That is, the optical black is lost in DNG, and part of the data is replaced with garbage (that's the right part of the uncompressed raw from A7S2)

 

[Iliah Borg]

I suspect what matters to 99.999% of us is whether the conversion to DNG preserves the image data

See the vertical band of duplicated pixels at the right edge of DNG:

DNG

source ARW
One may not realize the importance of the optical black for noise reduction, but surely added garbage raise certain concerns.

--

About LibRaw | LibRaw

www.libraw.org

 

[l_d_allan]

I suggest compressing them with a file archiver when archiving the files. 7-zip reduced the file size of my uncompressed files from today 57% which I expect is significantly better than you'll ever get with some computationally cheap on camera lossless compression (and those lossless compressed files are very hard to compress further). That the files are big while on the memory card or while on my HD doesn't bother me, just means I'll be archiving more often.

I think the issue is with post processing software not being able to directly access .zip's, .rar's, .lz's, etc.

Adobe does release new "Process Versions" from time to time ... 2003 -> 2010 -> 2012 -> ??? -> 2020. Each were MAJOR improvements. With CC, they are making incremental improvements, like the recent "Haze" and then "Haze" as a gradient/adjustment. At some point, I'd expect them to have something to compete with Dxo's fancy noise reduction.

It seems reasonable to revisit older photos when a new PV is released ... without having to decompress them to access.