How much disk space do I use in a year of hybrid camera use?

[jayboo]

This amateur managed to fill just under 3TB, stills only. Off to a heavy start, 1 Jan 100GB of RAWs but will be able to cull it when I get home. Ken

No where did I say stills only, you added that. Instead of trying to put in a cheap shot, why don't you reread the initial post I made.

Woah... ....back up. The way I read the comment was Gloomy1 was talking of his own experiences, why so touchy. Happy New Year.

Thanks for the clarity, I can see the other intention also. I had others assuming I was only using stills, so I thought this was yet another follow up on my saving 3TB of stills, and mistook the "This amateur" comment as pejorative. I am never to big to say I made a mistake.

Apologies to Gloomy1, if I misinterpreted your comment.

...and thanks to Jayboo for calling it out.

Aw.. well done you - we all make mistakes.

I was reading through the thread because I'm always interested how others treat their files and storage.

Myself, as a very enthusiastic hobbyist/amateur, I shoot every day, but I'm a pretty ferocious culler. Very aware that when I'm gone, they will most probably end up in land-fill. Anything I'm really enthusiastic about gets printed, either by myself or into books.

Again - Happy New Year--
Jayne

jayneboo

Explore jayneboo’s 7,974 photos on Flickr!

www.flickr.com

 

[jhunna]

This amateur managed to fill just under 3TB, stills only. Off to a heavy start, 1 Jan 100GB of RAWs but will be able to cull it when I get home. Ken

No where did I say stills only, you added that. Instead of trying to put in a cheap shot, why don't you reread the initial post I made.

Woah... ....back up. The way I read the comment was Gloomy1 was talking of his own experiences, why so touchy. Happy New Year.

Thanks for the clarity, I can see the other intention also. I had others assuming I was only using stills, so I thought this was yet another follow up on my saving 3TB of stills, and mistook the "This amateur" comment as pejorative. I am never to big to say I made a mistake.

Apologies to Gloomy1, if I misinterpreted your comment.

...and thanks to Jayboo for calling it out.

Aw.. well done you - we all make mistakes.

I was reading through the thread because I'm always interested how others treat their files and storage.

Myself, as a very enthusiastic hobbyist/amateur, I shoot every day, but I'm a pretty ferocious culler. Very aware that when I'm gone, they will most probably end up in land-fill. Anything I'm really enthusiastic about gets printed, either by myself or into books.

Again - Happy New Year--
Jayne

https://www.flickr.com/photos/jayneboo/

Thanks Jayne. I save everything, and then share things via google photos albums. If someone likes a picture, I go to my archives and either give them a better file, or print.

I haven't spoken much about printing, but that is my new love with this hobby.

 

[RunCatRun]

The terms "Compression" and "lossless" are not the same. I think it's obvious even to you. The topic starter jhunna is right. Sony is misleading consumers when they refer lossy formats as "lossless", claiming that they have just a different level of data compression. This is simply not true.

You can find many examples on YouTube and the Internet comparing the results of photos shot in different formats under controlled shooting conditions. And I will provide in this message with a lot of links to evidence to support my words as you did. :/

Yes. the "RAW L lossless" format is a good option. It's a compromise. I'm using virtualized NAS with a storage pool based on Intel D7-P5510 7.68TB, WD SN640 7.68TB for image processing and storage. However, it's quite expensive. I can't afford the "uncompressed" format. But I want to. It's a matter of difference in image quality.

"RAW L lossless" is not a true lossless format. If you don't trust YouTube, you can do your own tests to see for your self. But do you really have a qualification for it? It's not the same as simply screeming about the purity of Sony. Don't be a brainless fan as you are, Impulses, AlephNull and so on for now.

You don't need to respond, as this is more of a personal reflection for the forum members who are reading this thread casually. They should be aware of the facts.

 

[Impulses]

The terms "Compression" and "lossless" are not the same. I think it's obvious even to you. The topic starter jhunna is right. Sony is misleading consumers when they refer lossy formats as "lossless", claiming that they have just a different level of data compression. This is simply not true.

Incorrect.

You can find many examples on YouTube and the Internet comparing the results of photos shot in different formats under controlled shooting conditions. And I will provide in this message with a lot of links to evidence to support my words as you did. :/

Because they wouldn't stand up to scrutiny or because... ?

Yes. the "RAW L lossless" format is a good option. It's a compromise. I'm using virtualized NAS with a storage pool based on Intel D7-P5510 7.68TB, WD SN640 7.68TB for image processing and storage. However, it's quite expensive. I can't afford the "uncompressed" format. But I want to. It's a matter of difference in image quality.

"RAW L lossless" is not a true lossless format. If you don't trust YouTube, you can do your own tests to see for your self. But do you really have a qualification for it? It's not the same as simply screeming about the purity of Sony. Don't be a brainless fan as you are, Impulses, AlephNull and so on for now.

Ad hominem attacks, classy...

You don't need to respond, as this is more of a personal reflection for the forum members who are reading this thread casually. They should be aware of the facts.

Misinformation with no basis in reality != facts. I've never "screamed about the purity" of Sony and I don't trust their marketing one bit either, you're still grossly misunderstanding how lossless compression works, and then spreading FUD around it, right from your starting premise about the vocabulary itself. Expect pushback every time you go down this road, there's enough misinformation out there.

 

[RunCatRun]

You're right. There are lossless in the final image when shooting in the "Lossless Comp (L)" format. Don't be scared or confused by the hysteria of Sony fans.

The only thing I have a time to point out now is that it is necessary to ruthlessly delete photos of "controversial quality", as over time, the question of organizing a larger repository of "hot data" will arise. This means more controllers, more RAM, more CPU power, more cache disks, another unit size, file system, perfomance tests, RAID configuration and other issues.

And about cold backups. 1 SSD for a year of data. It's a 10 disks after 10 years of work. There are U.2 to USB adapters and U.2 drives in 15+TB. (LTO is for video only IMHO). It's more convinient and effective to analyze, store and evacuate (in case of emergency) only a couple of discs, rather then a pack of loose disks.

I sincerely wish you success in your work and business.

Ciao.

 

[jhunna]

You're right. There are lossless in the final image when shooting in the "Lossless Comp (L)" format. Don't be scared or confused by the hysteria of Sony fans.

The only thing I have a time to point out now is that it is necessary to ruthlessly delete photos of "controversial quality", as over time, the question of organizing a larger repository of "hot data" will arise. This means more controllers, more RAM, more CPU power, more cache disks, another unit size, file system, perfomance tests, RAID configuration and other issues.

And about cold backups. 1 SSD for a year of data. It's a 10 disks after 10 years of work. There are U.2 to USB adapters and U.2 drives in 15+TB. (LTO is for video only IMHO). It's more convinient and effective to analyze, store and evacuate (in case of emergency) only a couple of discs, rather then a pack of loose disks.

I sincerely wish you success in your work and business.

Ciao.

Thanks for the vote of confidence, it is appreciated.

 

[RunCatRun]

All you did in your messages was try to steer the conversation away from the "Sony camera data storage mode name not matching the actual result" issue to "what is lossless data compression" with actual errors.

About the method of cryptographic transformation of information in order to reduce the amount of information while maintaining quantity (lossless compression) I got acquainted from scientific articles and standards on specific algorithms, and not from the messages of another Sony fan. Thank you.

Sometimes you just have to admit that you made a mistake, apologize and move on.

 

[dazzerfong]

The terms "Compression" and "lossless" are not the same. I think it's obvious even to you. The topic starter jhunna is right. Sony is misleading consumers when they refer lossy formats as "lossless", claiming that they have just a different level of data compression. This is simply not true.

You can find many examples on YouTube and the Internet comparing the results of photos shot in different formats under controlled shooting conditions. And I will provide in this message with a lot of links to evidence to support my words as you did. :/

Yes. the "RAW L lossless" format is a good option. It's a compromise. I'm using virtualized NAS with a storage pool based on Intel D7-P5510 7.68TB, WD SN640 7.68TB for image processing and storage. However, it's quite expensive. I can't afford the "uncompressed" format. But I want to. It's a matter of difference in image quality.

"RAW L lossless" is not a true lossless format. If you don't trust YouTube, you can do your own tests to see for your self. But do you really have a qualification for it? It's not the same as simply screeming about the purity of Sony. Don't be a brainless fan as you are, Impulses, AlephNull and so on for now.

You don't need to respond, as this is more of a personal reflection for the forum members who are reading this thread casually. They should be aware of the facts.

Here we go again.....

1. Lossless compression is a type of compression, implying that no data was lost in the process. All zipped files on computers are lossless - you don't want your text in your zipped document to be garbled now do we?
2. Sony never misinterpreted their lossy compression as lossless. One poorly article implied their lossless as lossy, but that is demonstrably not true.
3. Raw lossless L's only compromise is that some programs which have not been updated to interpret it can't open said files as it's a completely different decoding system. Heck they haven't even updated their own pixel shift function to deal with it!
4. Yes I have done the tests before, ascertaining whether DR changes or if any compression artifacts showed up (they didnt). At least far better than those YouTube jokers who can't even do a controlled test to save their lives. I was a test engineer in a previous life.

 

[dazzerfong]

You're right. There are lossless in the final image when shooting in the "Lossless Comp (L)" format. Don't be scared or confused by the hysteria of Sony fans.

I'm confused - those 'Sony fans' were right (albeit unable to explain why), as opposed to OP who was wrong.

 

[jhunna]

1. Yes I have done the tests before, ascertaining whether DR changes or if any compression artifacts showed up (they didnt). At least far better than those YouTube jokers who can't even do a controlled test to save their lives. I was a test engineer in a previous life.

To be honest, DR changes or artifacts won't matter, because when the file is restored it should be an exact copy of the uncompressed file.

The question is when a lossless compressed file is restored is it an exact copy of the uncompressed file? And no one, not even Sony has shown that it is.

 

[dazzerfong]

1. Yes I have done the tests before, ascertaining whether DR changes or if any compression artifacts showed up (they didnt). At least far better than those YouTube jokers who can't even do a controlled test to save their lives. I was a test engineer in a previous life.

To be honest, DR changes or artifacts won't matter, because when the file is restored it should be an exact copy of the uncompressed file.

The question is when a lossless compressed file is restored is it an exact copy of the uncompressed file? And no one, not even Sony has shown that it is.

You still don't get it.

Lossy compression will incur unrecovereable losses in data in pursuit of efficiency. They're unreversible back to the original image, and the resulting image will never be identical to the original. I was trying to ascertain if lossless and uncompressed were functionally different (they weren't) - they had the same bit depth, same number of pixels and same DR with no lossy compression errors. I then dug into Rawtherapee (which is open source and therefore visible to everyone) and then finally determined it was actually lossless JPEG (more or less the same as Canon's implementation actually).

DR may be changed because on some cameras, using lossy drops it down from 14-bit acquisition to 12-bit. Technically if the sensor readout was 12-bit but no additional lossy compression was applied, it's still technically lossless.

The lossless compression instead does not incur data loss. It's a different way of packaging up the exact same data.

Sony doesn't need to show it, you can literally look at the data structure itself and see that it's a lossless jpeg92. Your technical illiteracy is not Sony's problem. Unless Sony is doing a lossy compression then a lossless compression on top (which is very stupid as it'll be slower to encode and actually take up more space than lossy for no reason), it's as good as proof that you're going to ever get.

If you want to test it yourself, get a random BMP image, save it in lossless JPEG and do a delta. I guarantee you it'll be the same. If not, go to the JPEG foundation and tell them their lossless compression scheme is actually lossy: might make a good buck outta them.

This is going around in circles: there's circumstancial evidence it's the same (ignoring poorly-conducted tests) and there's mathematical proof that it's lossless (i.e. identical to non-compressed after decompression). What more do you want? You want Mr Sony to come out and say lossless is actually lossless?

 

[Impulses]

All you did in your messages was try to steer the conversation away from the "Sony camera data storage mode name not matching the actual result" issue to "what is lossless data compression" with actual errors.

Feel free to link proof where lossless compression resulted in something not expected.

About the method of cryptographic transformation of information in order to reduce the amount of information while maintaining quantity (lossless compression) I got acquainted from scientific articles and standards on specific algorithms, and not from the messages of another Sony fan. Thank you.

Sometimes you just have to admit that you made a mistake, apologize and move on.

Sometimes you just have to admit you don't know what you're talking about, or show proof.

 

[jhunna]

1. Yes I have done the tests before, ascertaining whether DR changes or if any compression artifacts showed up (they didnt). At least far better than those YouTube jokers who can't even do a controlled test to save their lives. I was a test engineer in a previous life.

To be honest, DR changes or artifacts won't matter, because when the file is restored it should be an exact copy of the uncompressed file.

The question is when a lossless compressed file is restored is it an exact copy of the uncompressed file? And no one, not even Sony has shown that it is.

You still don't get it.

Lossy compression will incur unrecovereable losses in data in pursuit of efficiency. They're unreversible back to the original image, and the resulting image will never be identical to the original. I was trying to ascertain if lossless and uncompressed were functionally different (they weren't) - they had the same bit depth, same number of pixels and same DR with no lossy compression errors. I then dug into Rawtherapee (which is open source and therefore visible to everyone) and then finally determined it was actually lossless JPEG (more or less the same as Canon's implementation actually).

DR may be changed because on some cameras, using lossy drops it down from 14-bit acquisition to 12-bit. Technically if the sensor readout was 12-bit but no additional lossy compression was applied, it's still technically lossless.

The lossless compression instead does not incur data loss. It's a different way of packaging up the exact same data.

Sony doesn't need to show it, you can literally look at the data structure itself and see that it's a lossless jpeg92. Your technical illiteracy is not Sony's problem. Unless Sony is doing a lossy compression then a lossless compression on top (which is very stupid as it'll be slower to encode and actually take up more space than lossy for no reason), it's as good as proof that you're going to ever get.

No need to be nasty. You don't know my background, and I don't know yours other than a few words on a random forum. I follow that you are basing your entire premise on the fact that its a lossless jpeg92 file.

If you want to test it yourself, get a random BMP image, save it in lossless JPEG and do a delta. I guarantee you it'll be the same. If not, go to the JPEG foundation and tell them their lossless compression scheme is actually lossy: might make a good buck outta them.

This is not a test for Sony's lossless compression. The only files you need are two files taken by the same camera in the exact same condition. Extract the lossless compressed file and it should be identical. If it isn't then there was a change and/or a loss.

This is going around in circles: there's circumstancial evidence it's the same (ignoring poorly-conducted tests) and there's mathematical proof that it's lossless (i.e. identical to non-compressed after decompression). What more do you want? You want Mr Sony to come out and say lossless is actually lossless?

Where is the mathematical proof that its identical to the non-compressed file after decompression? Like I said no one has provided that...

 

[Impulses]

1. Yes I have done the tests before, ascertaining whether DR changes or if any compression artifacts showed up (they didnt). At least far better than those YouTube jokers who can't even do a controlled test to save their lives. I was a test engineer in a previous life.

To be honest, DR changes or artifacts won't matter, because when the file is restored it should be an exact copy of the uncompressed file.

The question is when a lossless compressed file is restored is it an exact copy of the uncompressed file? And no one, not even Sony has shown that it is.

You still don't get it.

Lossy compression will incur unrecovereable losses in data in pursuit of efficiency. They're unreversible back to the original image, and the resulting image will never be identical to the original. I was trying to ascertain if lossless and uncompressed were functionally different (they weren't) - they had the same bit depth, same number of pixels and same DR with no lossy compression errors. I then dug into Rawtherapee (which is open source and therefore visible to everyone) and then finally determined it was actually lossless JPEG (more or less the same as Canon's implementation actually).

DR may be changed because on some cameras, using lossy drops it down from 14-bit acquisition to 12-bit. Technically if the sensor readout was 12-bit but no additional lossy compression was applied, it's still technically lossless.

The lossless compression instead does not incur data loss. It's a different way of packaging up the exact same data.

Sony doesn't need to show it, you can literally look at the data structure itself and see that it's a lossless jpeg92. Your technical illiteracy is not Sony's problem. Unless Sony is doing a lossy compression then a lossless compression on top (which is very stupid as it'll be slower to encode and actually take up more space than lossy for no reason), it's as good as proof that you're going to ever get.

No need to be nasty. You don't know my background, and I don't know yours other than a few words on a random forum. I follow that you are basing your entire premise on the fact that its a lossless jpeg92 file.

If you want to test it yourself, get a random BMP image, save it in lossless JPEG and do a delta. I guarantee you it'll be the same. If not, go to the JPEG foundation and tell them their lossless compression scheme is actually lossy: might make a good buck outta them.

This is not a test for Sony's lossless compression. The only files you need are two files taken by the same camera in the exact same condition. Extract the lossless compressed file and it should be identical. If it isn't then there was a change and/or a loss.

This is going around in circles: there's circumstancial evidence it's the same (ignoring poorly-conducted tests) and there's mathematical proof that it's lossless (i.e. identical to non-compressed after decompression). What more do you want? You want Mr Sony to come out and say lossless is actually lossless?

Where is the mathematical proof that its identical to the non-compressed file after decompression? Like I said no one has provided that...

No one has provided proof to the contrary either.

 

[dazzerfong]

1. Yes I have done the tests before, ascertaining whether DR changes or if any compression artifacts showed up (they didnt). At least far better than those YouTube jokers who can't even do a controlled test to save their lives. I was a test engineer in a previous life.

To be honest, DR changes or artifacts won't matter, because when the file is restored it should be an exact copy of the uncompressed file.

The question is when a lossless compressed file is restored is it an exact copy of the uncompressed file? And no one, not even Sony has shown that it is.

You still don't get it.

Lossy compression will incur unrecovereable losses in data in pursuit of efficiency. They're unreversible back to the original image, and the resulting image will never be identical to the original. I was trying to ascertain if lossless and uncompressed were functionally different (they weren't) - they had the same bit depth, same number of pixels and same DR with no lossy compression errors. I then dug into Rawtherapee (which is open source and therefore visible to everyone) and then finally determined it was actually lossless JPEG (more or less the same as Canon's implementation actually).

DR may be changed because on some cameras, using lossy drops it down from 14-bit acquisition to 12-bit. Technically if the sensor readout was 12-bit but no additional lossy compression was applied, it's still technically lossless.

The lossless compression instead does not incur data loss. It's a different way of packaging up the exact same data.

Sony doesn't need to show it, you can literally look at the data structure itself and see that it's a lossless jpeg92. Your technical illiteracy is not Sony's problem. Unless Sony is doing a lossy compression then a lossless compression on top (which is very stupid as it'll be slower to encode and actually take up more space than lossy for no reason), it's as good as proof that you're going to ever get.

No need to be nasty. You don't know my background, and I don't know yours other than a few words on a random forum. I follow that you are basing your entire premise on the fact that its a lossless jpeg92 file.

If you want to test it yourself, get a random BMP image, save it in lossless JPEG and do a delta. I guarantee you it'll be the same. If not, go to the JPEG foundation and tell them their lossless compression scheme is actually lossy: might make a good buck outta them.

This is not a test for Sony's lossless compression. The only files you need are two files taken by the same camera in the exact same condition. Extract the lossless compressed file and it should be identical. If it isn't then there was a change and/or a loss.

This is going around in circles: there's circumstancial evidence it's the same (ignoring poorly-conducted tests) and there's mathematical proof that it's lossless (i.e. identical to non-compressed after decompression). What more do you want? You want Mr Sony to come out and say lossless is actually lossless?

Where is the mathematical proof that its identical to the non-compressed file after decompression? Like I said no one has provided that...

I literally gave you proof, and you didn't even recognise it as such. You don't have to take my word that it's an ljpeg92 encoding, I linked it on Rawtherapee's repo!

You literally cannot do that test you proposed because the sensor will always have random noise, thus two pictures (even if uncompressed) will literally never have the same raw acquisition.

Proof is there, again, not my problem or Sony's you don't understand it. Here it is again: https://github.com/Beep6581/RawTherapee/blob/dev/rtengine/libraw/src/decoders/sonycc.cpp

Or if you want it straight from LibRaw (which is where RT poached it from):

https://github.com/LibRaw/LibRaw/blob/master/src/decoders/sonycc.cpp

If you want more proof, Darktable also supports it through their rawspeed library: this one's a little bit easier to explain. See for their ARW decoder, look at line 178 and follow from there: it calls the DecodeLJpeg function which then, after scraping all the info necessary, passes it to LJpegDecoder in line 381. There, in line 101 it calls AbstractLJpegDecoder which is what actually decodes it.

 

[jhunna]

1. Yes I have done the tests before, ascertaining whether DR changes or if any compression artifacts showed up (they didnt). At least far better than those YouTube jokers who can't even do a controlled test to save their lives. I was a test engineer in a previous life.

To be honest, DR changes or artifacts won't matter, because when the file is restored it should be an exact copy of the uncompressed file.

The question is when a lossless compressed file is restored is it an exact copy of the uncompressed file? And no one, not even Sony has shown that it is.

You still don't get it.

Lossy compression will incur unrecovereable losses in data in pursuit of efficiency. They're unreversible back to the original image, and the resulting image will never be identical to the original. I was trying to ascertain if lossless and uncompressed were functionally different (they weren't) - they had the same bit depth, same number of pixels and same DR with no lossy compression errors. I then dug into Rawtherapee (which is open source and therefore visible to everyone) and then finally determined it was actually lossless JPEG (more or less the same as Canon's implementation actually).

DR may be changed because on some cameras, using lossy drops it down from 14-bit acquisition to 12-bit. Technically if the sensor readout was 12-bit but no additional lossy compression was applied, it's still technically lossless.

The lossless compression instead does not incur data loss. It's a different way of packaging up the exact same data.

Sony doesn't need to show it, you can literally look at the data structure itself and see that it's a lossless jpeg92. Your technical illiteracy is not Sony's problem. Unless Sony is doing a lossy compression then a lossless compression on top (which is very stupid as it'll be slower to encode and actually take up more space than lossy for no reason), it's as good as proof that you're going to ever get.

No need to be nasty. You don't know my background, and I don't know yours other than a few words on a random forum. I follow that you are basing your entire premise on the fact that its a lossless jpeg92 file.

If you want to test it yourself, get a random BMP image, save it in lossless JPEG and do a delta. I guarantee you it'll be the same. If not, go to the JPEG foundation and tell them their lossless compression scheme is actually lossy: might make a good buck outta them.

This is not a test for Sony's lossless compression. The only files you need are two files taken by the same camera in the exact same condition. Extract the lossless compressed file and it should be identical. If it isn't then there was a change and/or a loss.

This is going around in circles: there's circumstancial evidence it's the same (ignoring poorly-conducted tests) and there's mathematical proof that it's lossless (i.e. identical to non-compressed after decompression). What more do you want? You want Mr Sony to come out and say lossless is actually lossless?

Where is the mathematical proof that its identical to the non-compressed file after decompression? Like I said no one has provided that...

I literally gave you proof, and you didn't even recognise it as such. You don't have to take my word that it's an ljpeg92 encoding, I linked it on Rawtherapee's repo!

I never refuted if it was a lsjpeg92 file. I picked that up the first time you stated it.

You literally cannot do that test you proposed because the sensor will always have random noise, thus two pictures (even if uncompressed) will literally never have the same raw acquisition.

Ok, that's a good point, but it will speak to the ability to test ANYTHING regarding photography files if there is random noise. The best we can do is take the same picture and the same file under the same conditions, and then compare. At a minimum, the uncompressed files should be extremely close in character, size, etc...

Proof is there, again, not my problem or Sony's you don't understand it. Here it is again: https://github.com/Beep6581/RawTherapee/blob/dev/rtengine/libraw/src/decoders/sonycc.cpp

Again, I didn't refute this.

Or if you want it straight from LibRaw (which is where RT poached it from):

https://github.com/LibRaw/LibRaw/blob/master/src/decoders/sonycc.cpp

I told you I started doing the decompression process last night, but don't have a c compiler to finish the compiling until I build a linux environment and can use gcc. And the complied windows versions like DCRaw don't recognize the the Sony file type.

That said, I have enough information, to do what I need to to prove the veracity of the lossless moniker for the raw files.

 

[jhunna]

Where is the mathematical proof that its identical to the non-compressed file after decompression? Like I said no one has provided that...

I literally gave you proof, and you didn't even recognise it as such. You don't have to take my word that it's an ljpeg92 encoding, I linked it on Rawtherapee's repo!

You literally cannot do that test you proposed because the sensor will always have random noise, thus two pictures (even if uncompressed) will literally never have the same raw acquisition.

Proof is there, again, not my problem or Sony's you don't understand it. Here it is again: https://github.com/Beep6581/RawTherapee/blob/dev/rtengine/libraw/src/decoders/sonycc.cpp

Or if you want it straight from LibRaw (which is where RT poached it from):

https://github.com/LibRaw/LibRaw/blob/master/src/decoders/sonycc.cpp

If you want more proof, Darktable also supports it through their rawspeed library: this one's a little bit easier to explain. See for their ARW decoder, look at line 178 and follow from there: it calls the DecodeLJpeg function which then, after scraping all the info necessary, passes it to LJpegDecoder in line 381. There, in line 101 it calls AbstractLJpegDecoder which is what actually decodes it.

Again thanks for this, if nothing else it confirms we are seeing the same thing to this point. I prefer to decode the files and work with them in cli to so I can see all the backend work that's being performed and or do additionally logging if possible. Libraw should be all I need ...

 

[dazzerfong]

1. Yes I have done the tests before, ascertaining whether DR changes or if any compression artifacts showed up (they didnt). At least far better than those YouTube jokers who can't even do a controlled test to save their lives. I was a test engineer in a previous life.

To be honest, DR changes or artifacts won't matter, because when the file is restored it should be an exact copy of the uncompressed file.

The question is when a lossless compressed file is restored is it an exact copy of the uncompressed file? And no one, not even Sony has shown that it is.

You still don't get it.

Lossy compression will incur unrecovereable losses in data in pursuit of efficiency. They're unreversible back to the original image, and the resulting image will never be identical to the original. I was trying to ascertain if lossless and uncompressed were functionally different (they weren't) - they had the same bit depth, same number of pixels and same DR with no lossy compression errors. I then dug into Rawtherapee (which is open source and therefore visible to everyone) and then finally determined it was actually lossless JPEG (more or less the same as Canon's implementation actually).

DR may be changed because on some cameras, using lossy drops it down from 14-bit acquisition to 12-bit. Technically if the sensor readout was 12-bit but no additional lossy compression was applied, it's still technically lossless.

The lossless compression instead does not incur data loss. It's a different way of packaging up the exact same data.

Sony doesn't need to show it, you can literally look at the data structure itself and see that it's a lossless jpeg92. Your technical illiteracy is not Sony's problem. Unless Sony is doing a lossy compression then a lossless compression on top (which is very stupid as it'll be slower to encode and actually take up more space than lossy for no reason), it's as good as proof that you're going to ever get.

No need to be nasty. You don't know my background, and I don't know yours other than a few words on a random forum. I follow that you are basing your entire premise on the fact that its a lossless jpeg92 file.

If you want to test it yourself, get a random BMP image, save it in lossless JPEG and do a delta. I guarantee you it'll be the same. If not, go to the JPEG foundation and tell them their lossless compression scheme is actually lossy: might make a good buck outta them.

This is not a test for Sony's lossless compression. The only files you need are two files taken by the same camera in the exact same condition. Extract the lossless compressed file and it should be identical. If it isn't then there was a change and/or a loss.

This is going around in circles: there's circumstancial evidence it's the same (ignoring poorly-conducted tests) and there's mathematical proof that it's lossless (i.e. identical to non-compressed after decompression). What more do you want? You want Mr Sony to come out and say lossless is actually lossless?

Where is the mathematical proof that its identical to the non-compressed file after decompression? Like I said no one has provided that...

I literally gave you proof, and you didn't even recognise it as such. You don't have to take my word that it's an ljpeg92 encoding, I linked it on Rawtherapee's repo!

I never refuted if it was a lsjpeg92 file. I picked that up the first time you stated it.

You literally cannot do that test you proposed because the sensor will always have random noise, thus two pictures (even if uncompressed) will literally never have the same raw acquisition.

Ok, that's a good point, but it will speak to the ability to test ANYTHING regarding photography files if there is random noise. The best we can do is take the same picture and the same file under the same conditions, and then compare. At a minimum, the uncompressed files should be extremely close in character, size, etc...

Proof is there, again, not my problem or Sony's you don't understand it. Here it is again: https://github.com/Beep6581/RawTherapee/blob/dev/rtengine/libraw/src/decoders/sonycc.cpp

Again, I didn't refute this.

Or if you want it straight from LibRaw (which is where RT poached it from):

https://github.com/LibRaw/LibRaw/blob/master/src/decoders/sonycc.cpp

I told you I started doing the decompression process last night, but don't have a c compiler to finish the compiling until I build a linux environment and can use gcc. And the complied windows versions like DCRaw don't recognize the the Sony file type.

That said, I have enough information, to do what I need to to prove the veracity of the lossless moniker for the raw files.

If you're doing a mathematical proof, you literally cannot use the sensor as there's always going to be randomness, so a true proof cannot be done there like that. You can verify for functional differences through photos (i.e. are there any practical differences), but not mathematical differences (which is what you were demanding).

Dcraw hasn't been updated in a very long time, well before the introduction of lossless ARWs so I'm not surprised it doesn't work.

How do you intend to prove that lossless raws are indeed lossless by compiling it yourself? Are you going to confirm Libraw's ability to load the image, and only when you actually compile it yourself from said source code, that the implementation is valid?

For what it's worth, even between uncompressed and lossy raw, the differences are so tiny you won't be able to see it unless you start forcing posterisation errors with some pretty extreme editing. You are never going to get images that are somehow magically blurrier because of a different compression scheme.

 

[dazzerfong]

Where is the mathematical proof that its identical to the non-compressed file after decompression? Like I said no one has provided that...

I literally gave you proof, and you didn't even recognise it as such. You don't have to take my word that it's an ljpeg92 encoding, I linked it on Rawtherapee's repo!

You literally cannot do that test you proposed because the sensor will always have random noise, thus two pictures (even if uncompressed) will literally never have the same raw acquisition.

Proof is there, again, not my problem or Sony's you don't understand it. Here it is again: https://github.com/Beep6581/RawTherapee/blob/dev/rtengine/libraw/src/decoders/sonycc.cpp

Or if you want it straight from LibRaw (which is where RT poached it from):

https://github.com/LibRaw/LibRaw/blob/master/src/decoders/sonycc.cpp

If you want more proof, Darktable also supports it through their rawspeed library: this one's a little bit easier to explain. See for their ARW decoder, look at line 178 and follow from there: it calls the DecodeLJpeg function which then, after scraping all the info necessary, passes it to LJpegDecoder in line 381. There, in line 101 it calls AbstractLJpegDecoder which is what actually decodes it.

Again thanks for this, if nothing else it confirms we are seeing the same thing to this point. I prefer to decode the files and work with them in cli to so I can see all the backend work that's being performed and or do additionally logging if possible. Libraw should be all I need ...

Curious to see how you're going to confirm the results now..........

 

[jhunna]

Where is the mathematical proof that its identical to the non-compressed file after decompression? Like I said no one has provided that...

I literally gave you proof, and you didn't even recognise it as such. You don't have to take my word that it's an ljpeg92 encoding, I linked it on Rawtherapee's repo!

You literally cannot do that test you proposed because the sensor will always have random noise, thus two pictures (even if uncompressed) will literally never have the same raw acquisition.

Proof is there, again, not my problem or Sony's you don't understand it. Here it is again: https://github.com/Beep6581/RawTherapee/blob/dev/rtengine/libraw/src/decoders/sonycc.cpp

Or if you want it straight from LibRaw (which is where RT poached it from):

https://github.com/LibRaw/LibRaw/blob/master/src/decoders/sonycc.cpp

If you want more proof, Darktable also supports it through their rawspeed library: this one's a little bit easier to explain. See for their ARW decoder, look at line 178 and follow from there: it calls the DecodeLJpeg function which then, after scraping all the info necessary, passes it to LJpegDecoder in line 381. There, in line 101 it calls AbstractLJpegDecoder which is what actually decodes it.

Again thanks for this, if nothing else it confirms we are seeing the same thing to this point. I prefer to decode the files and work with them in cli to so I can see all the backend work that's being performed and or do additionally logging if possible. Libraw should be all I need ...

Curious to see how you're going to confirm the results now..........

To be honest I am not sure it is worth sharing because, something regarding my methodology will be attacked or criticized. The info you have shared about the predictive encryption helped a lot with how I was looking at this.

A lot of this is confusion around terms and definitions, and I don't want to argue that. I would just rather look at the data, and see what Sony is doing to the files...

Not that it matters I just want to know...