New version of JPEG developed in Leipzig

[ApertureAcolyte]

These are the exact same arguments we heard when dng was introduced. My objection isn't to the concept of what is occurring so much as it is calling it jpg. THAT is pure marketing duplicity.

Personally, if we can come up with a "universal" raw format that can be supported by more than Adobe I would be happy. But from what I've read so far, this "ain't it".

-

It's to get dumb people to use a lossless format

This has nothing to do with raw.

 

[Allan Olesen]

The raw data in Canon CR2 files are compressed with lossless JPEG. I think Adobe's DNG are too, but I am not sure.

I've always understood that Canon raw files are based on a modified tiff format (16-bit).

As has already been pointed out, jpg is only 8-bit whereas most recent raw formats are at least 14-bit.

It does not surprise me that someone says I am wrong, based on what they have always understood.

The raw data in Canon CR2 files are compressed with lossless JPEG. I have no idea why you think that lossless JPEG is 8 bit.

You can read a detailed description of what a .CR2 contains here:

http://lclevy.free.fr/cr2/#lossless

 

[GodSpeaks]

It is still an image format. RAW is not an image format, but sensor data that still requires interpretation (raw converter) to get it into an image format.

So I just cannot ever see it replacing raw.

 

[GodSpeaks]

So what you're saying is when Adobe updates Adobe Raw that even my D700 may have a better way to perform this transformation in software.

Yes, but not just Adobe, all raw converters and even new converters. As the raw conversion software is updated, the algorythms used to do the conversions are changed, updated/improved, even replaced with new algorythms. What this means is that the ability of the converter to extract additional or better detail may improve as with the ability to render colors better.

I have raw files from the early 2000's which when converted today with modern current converters are hugely improved over the converters used at the time.

I would expect that improvements made today are much more incremental than what was happening 10 plus years ago.

 

[robert1955]

Hi,

Has anyone heard about a new version of JPEG which is a lossless format and could replace RAW? It seems to be available as an open source project, but I don't know whether it can be used with the current range of editing software.

anyone want to replace RAW?

One reason is to eliminate the need of upgrading software to process the file.

If you are using compatible software you never need to upgrade that software to process the file. Only the people that are constantly buying cameras need to worry about such things.

I'm not trying to be a wise guy, I'm trying to understand raw. Why, when a new camera comes out, the raw converter needs to be updated? Are you saying in effect a nef file from a D3 is different than an nef file from a D700? Or a nef file from a D4 is different than a nef file from a Df? If so, why?

The raw file contains the data derived from the electron charges collected on the sensor during the exposure. Different sensors respond differently to the light, and even the same sensor in different cameras can have different AA filters and other elements that make it respond differently. The information on the sensor is then transformed by the read circuitry, the analog amplifiers, the ADC (analog to digital converter), the digital software into the final raw data. All these elements can and do differ from camera to camera. The D300, for example, has a different NEF from the D300s.

It is the job of the raw converter to take these raw data (separate so-called R, G, and B channels) and interpolate (demosaic) them into an image with color RGB pixels. Any differences in any of the elements noted above will require different behavior by the raw converter. The manufacturer's proprietary software, of course, knows just how to do this for each of their cameras. Third-party raw converters must "reverse engineer" the behavior of each new camera to determine just what it is doing.

 

[canonagain123]

These are the exact same arguments we heard when dng was introduced. My objection isn't to the concept of what is occurring so much as it is calling it jpg. THAT is pure marketing duplicity.

Personally, if we can come up with a "universal" raw format that can be supported by more than Adobe I would be happy. But from what I've read so far, this "ain't it".

 

[CAcreeks]

http://www.popphoto.com/news/2014/01/jpeg-standard-91-will-bring-12-bit-color-lossless-compression

I thought you were talking about JPEG 2000, which (it would be fair to say) has been a flop.

The IJG (independent JPEG group) is highly respected and has done great work in the past. So I hope this effort succeeds.

As I have posted many times before, Raw would be totally unnecessary if cameras produced a lossless editable image.

P.S. Ironic that Pop Photo has news. I almost don't believe it!

 

[BG454]

The raw data in Canon CR2 files are compressed with lossless JPEG. I have no idea why you think that lossless JPEG is 8 bit.

You can read a detailed description of what a .CR2 contains here:

http://lclevy.free.fr/cr2/#lossless

Ah yes, in which it clearly states (under section 2.1 Overview):

"The .CR2 file is based on the TIFF file format. This TIFF file has 4 Image File Directories (IFDs). "

Sites which write "RAW" in capital letters, as if it's an acronym, annoy me immensely.

 

[richardplondon]

So what you're saying is when Adobe updates Adobe Raw that even my D700 may have a better way to perform this transformation in software.

Of course, the camera itself will not change just because of what Adobe does - though firmware updates can certainly do that when it comes to in-camera JPG.

If I understand your question correctly, the answer is "no".

When Adobe updates its software, it will typically only include the ability to "understand" the raw file parameters in raw file versions produced by new camera models, it will not, AFAIK, change how the raw file parameters in the raw file versions produced by previously existing camera models are interpreted and processed.

Well, precisely this DID happen with the D7000, and with other models too IIRC. The early m43 support for lens correction has changed beyond recognition. The Fuji non-standard sensor patterns are AFAIK much better interpreted now.

An only-just-released camera model may get just "preliminary" support from Adobe due to lack of time, that is then improved in the nezt release of ACR / Lightroom - e.g. adding a selection of camera matching calibration profiles, or more basically, achieving a better-tuned demosaic. Also Adobe may come back and supersede poor camera profiles, with "improved" ones - either camera-matching, or generic Adobe Standard, to resolve specific issues.

And in general, over time, even for a camera that is not newly introduced, the progressive changes in noise reduction, sharpening, tonal controls etc as well as the extended features such as local adjustment and cloning, also availability of lens profiles, add up to improved results (leaving aside any pixel / layer-based postprocessing).

 

[Ray Ritchie]

Has anyone heard about a new version of JPEG which is a lossless format and could replace RAW?

RAW files are NOT image formats. They cannot be sensibly displayed as an image. They store sensor data, which has to be converted into a format ( like a JPEG ) which can be viewed normally. This process involves processing steps.

I hear this sort of characterization of raw a lot, but the importance of the point is lost on me. Perhaps you could elaborate on what constitutes an "image format"?

Neither a JPEG file nor a raw file can be displayed directly, without the aid of some decoding software. The software for decoding a JPEG is actually pretty complex, since the encoding process uses several layers of different techniques, including segmenting the image into blocks, differential encoding, transformation to the frequency domain, Huffman encoding, etc. In some ways, typical raw formats are more directly related to the original image, in that most of the bits in a raw format have a direct relationship to a known portion of the image as it was focussed on the sensor by the lens. In the case of JPEG, I don't think you can point to any particular bit or byte in the JPEG-encoded file which relates directly to the original image seen by the sensor.

So since neither JPEG nor raw files can be directly displayed, and the JPEG is derived from the raw, and both require complex decoding software, why is one more of an "image format" than the other?

JPEG is a format designed to store an image. You cannot recover a RAW file from a JPEG ( or any "cooked" image format ). This is because the processing that converts the sensor data in the RAW file make irreversible changes to the data.

There is a lossless JPEG format called JPEG2000. This is not normally used in photographic or media applications. It is used in some medical systems ( where any loss of data is unacceptable ). If JPEG2000 did not catch on ( and it has 16-bit capability, not 12 ) it's seems unlikely that this new format will.

I'm not sure how you can make that judgment without a detailed study of the new proposal.
Ray
My blog: http://www.rritchie.com/wordpress
My Flickr page: http://www.flickr.com/photos/rritchie/

 

[NJOW]

As I understand it, a raw file is indeed data encoded by a specified process (algorithm), and to that extent it is an image format.

The key difference is that this "format" varies from camera to camera (or at least from brand to brand) and is a generic term for "uncooked" files from a camera, not an individual format itself, even if the file is based on, in whole or part, an image format.

This is why it's "raw" and not "RAW" as so many people erroneously use (especially here, even, I've noticed from people claiming to be photography teachers). This may be slight pedantry, but it is somewhat important given that some people appear confused as to what raw files are.

RAW does not stand for anything, it is not an acronym - I know you've used it correctly, so I'm not directing this at you - however given the substance of your post and the post you replied to, I would have expected you to have said "RAW" and him/her to say "raw" ;-)

 

[Allan Olesen]

The raw data in Canon CR2 files are compressed with lossless JPEG. I have no idea why you think that lossless JPEG is 8 bit.
You can read a detailed description of what a .CR2 contains here:

http://lclevy.free.fr/cr2/#lossless

Ah yes, in which it clearly states (under section 2.1 Overview):

"The .CR2 file is based on the TIFF file format. This TIFF file has 4 Image File Directories (IFDs). "

How is that relevant? It does not disprove that CR2 is compressed with lossless JPEG, since TIFF is a container format which can be used for several different compressed and non-compressed image formats.

Sites which write "RAW" in capital letters, as if it's an acronym, annoy me immensely.

Me too, but how is that relevant?

 

[darklamp]

Has anyone heard about a new version of JPEG which is a lossless format and could replace RAW?

RAW files are NOT image formats. They cannot be sensibly displayed as an image. They store sensor data, which has to be converted into a format ( like a JPEG ) which can be viewed normally. This process involves processing steps.

I hear this sort of characterization of raw a lot, but the importance of the point is lost on me. Perhaps you could elaborate on what constitutes an "image format"?

A RAW file contains photon counts. A JPEG, TIFF or other image format does not.

I can create an arbitrary number of images from a single RAW. Any one of those is a legitimate image of the scene which is defined by a combination of the original RAW data and parameters I add in processing from RAW to JPEG. These parameters control things like tone curve ( RAW data does not have a luminance curve that is a match to human vision and the media we design for that purpose.

I cannot display an image without adding some parameters to the RAW data to control that processing.

A JPEG or other image format already has those parameters "cooked in". The JPEG stores data that matches the display requirements needed to produce an image. The data in a JPEG is designed to be used by output devices. RAW data is not.

Neither a JPEG file nor a raw file can be displayed directly, without the aid of some decoding software. The software for decoding a JPEG is actually pretty complex, since the encoding process uses several layers of different techniques, including segmenting the image into blocks, differential encoding, transformation to the frequency domain, Huffman encoding, etc.

You are confusing compression of data ( which is what JPEG does ) and the transformation processing required to convert photon counts to output device orientated display data.

Actually some RAW files use compression not dissimilar to JPEG encoding.

From the point of view of a computer engineer compression is seen as a separate thing from any other kinds of processing. The difference between the formats remains - an image format is "display ready" data ( compressed or not ) and a RAW file does NOT contain display ready data ( compressed or not ).

In some ways, typical raw formats are more directly related to the original image, in that most of the bits in a raw format have a direct relationship to a known portion of the image as it was focussed on the sensor by the lens. In the case of JPEG, I don't think you can point to any particular bit or byte in the JPEG-encoded file which relates directly to the original image seen by the sensor.

Actually that's the point. The image data is processed data intended for consumption by output media. It does not require any relationship with the source data ( the RAW data ). It can be anything.

So since neither JPEG nor raw files can be directly displayed, and the JPEG is derived from the raw, and both require complex decoding software, why is one more of an "image format" than the other?

The data in a RAW file cannot be displayed with additional processing controlled by parameters defined outside the RAW file.

The data in an image format ( like JPEG ) can be displayed without reference to data outside the scope of the image file's own data.

JPEG is a format designed to store an image. You cannot recover a RAW file from a JPEG ( or any "cooked" image format ). This is because the processing that converts the sensor data in the RAW file make irreversible changes to the data.

There is a lossless JPEG format called JPEG2000. This is not normally used in photographic or media applications. It is used in some medical systems ( where any loss of data is unacceptable ). If JPEG2000 did not catch on ( and it has 16-bit capability, not 12 ) it's seems unlikely that this new format will.

I'm not sure how you can make that judgment without a detailed study of the new proposal.

I can. I've been involved in both IT and photography for a long, long time now.

You need to understand that the reason why the existing 8-bit 3-channel JPEG format has survived and will continue to survive as the primary format for processed image exchange is that it is a format that is used everywhere. It's good enough for the job it does and converting or replacing systems to use a new standard would be so expensive that it's prohibitive.

Standards persist because they work well enough and the cost of change is prohibitive for the marginal benefit.

That's why this new JPEG format won't set the world alight.

 

[canonagain123]

How is that relevant? It does not disprove that CR2 is compressed with lossless JPEG, since TIFF is a container format which can be used for several different compressed and non-compressed image formats.

Sites which write "RAW" in capital letters, as if it's an acronym, annoy me immensely.

Me too, but how is that relevant?

there is no 14-bit JFIF standard, therefore the 14-bit data cannot be stored as compressed, uncompressed, encoded, unencoded, encrypted or unencrypted JPG anything. Was that clear enough for you, or do you need it phrased differently?

I'm saying 14-bit, because RAW images out of most cameras are 14-bit, but there are also 16-bit raw images, 24-bit, 32-bit, 48-bit and 64-bit raw images. There is no way to store any of those in an 8-bit container. The 12-bit JPG standard is not widely supported, and is not used outside medical imaging where the 12 bits are used to represent different shades of gray.

 

[Glen Barrington]

How is that relevant? It does not disprove that CR2 is compressed with lossless JPEG, since TIFF is a container format which can be used for several different compressed and non-compressed image formats.

Sites which write "RAW" in capital letters, as if it's an acronym, annoy me immensely.

Me too, but how is that relevant?

there is no 14-bit JFIF standard, therefore the 14-bit data cannot be stored as compressed, uncompressed, encoded, unencoded, encrypted or unencrypted JPG anything. Was that clear enough for you, or do you need it phrased differently?

I'm saying 14-bit, because RAW images out of most cameras are 14-bit, but there are also 16-bit raw images, 24-bit, 32-bit, 48-bit and 64-bit raw images. There is no way to store any of those in an 8-bit container. The 12-bit JPG standard is not widely supported, and is not used outside medical imaging where the 12 bits are used to represent different shades of gray.

Would there be as many as 50 shades of gray?

 

[newmikey]

Has anyone heard about a new version of JPEG which is a lossless format and could replace RAW? It seems to be available as an open source project, but I don't know whether it can be used with the current range of editing software.

Lots of misinterpretations in this thread so allow me to make an attempt to clear a little of the fog:

1. Yes, the new libjpeg library has lossless, 12-bit capability

2. No image format can replace RAW so neither can libjpeg

3. As a lot of commercial editing software already uses libjpeg, you will see it integrated in all commercial software over the next few years, however accessibility of its features will depend on the software developers putting the required controls in their GUI's .

4. As all of the open source editors use libjpeg and the open-source community is generally quick to integrate new features, you will see existing editors (and raw converters) gain access to the new features pretty quick. This will allow for better preservation of detail and color during processing, preventing histogram spikes, posterization etc. when saving images.

5. As almost all webservers use libjpeg to display jpeg images, you should see no issues in websites quickly allowing for the new capabilities and displaying these 12-bits lossless jpegs. (but...see (6) it will not matter that much)

6. As almost all computer CRT's and LCD's have limited display capabilities, you will not see a lot of benefits when displaying these images on-screen either directly or through a website.

7. As most RIP's (Raster Image Processors) use libjpeg, you will probably see some benefit in professionally printed output, possibly eventually also through better printerdrivers in your own high-end inkjets.

8. As this concerns the integration of new capabilities in an already widespread library, there will be no issues like the ones with JPEG2000 which was entirely different software.

9. It will take a long time for the new capabilities of libjpeg to make their way into camera firmware (if ever) due to memory and processing speed requirements of manipulating thàt much more data with a small processor and limited memoryspace in the camera hardware.

My personal opinion? Great if you shoot raw, convert to jpeg using the new capabilities with the intention to print. No huge impact when shooting/processing for web or OOC straight jpegs.

--
http://mike.bing-photography.com
http://www.flickr.com/photos/newmikey/
Opensource: fresher, tastier and cheaper!

 

[knickerhawk]

Has anyone heard about a new version of JPEG which is a lossless format and could replace RAW? It seems to be available as an open source project, but I don't know whether it can be used with the current range of editing software.

Lots of misinterpretations in this thread so allow me to make an attempt to clear a little of the fog:

1. Yes, the new libjpeg library has lossless, 12-bit capability

2. No image format can replace RAW so neither can libjpeg

3. As a lot of commercial editing software already uses libjpeg, you will see it integrated in all commercial software over the next few years, however accessibility of its features will depend on the software developers putting the required controls in their GUI's .

4. As all of the open source editors use libjpeg and the open-source community is generally quick to integrate new features, you will see existing editors (and raw converters) gain access to the new features pretty quick. This will allow for better preservation of detail and color during processing, preventing histogram spikes, posterization etc. when saving images.

5. As almost all webservers use libjpeg to display jpeg images, you should see no issues in websites quickly allowing for the new capabilities and displaying these 12-bits lossless jpegs. (but...see (6) it will not matter that much)

6. As almost all computer CRT's and LCD's have limited display capabilities, you will not see a lot of benefits when displaying these images on-screen either directly or through a website.

7. As most RIP's (Raster Image Processors) use libjpeg, you will probably see some benefit in professionally printed output, possibly eventually also through better printerdrivers in your own high-end inkjets.

8. As this concerns the integration of new capabilities in an already widespread library, there will be no issues like the ones with JPEG2000 which was entirely different software.

9. It will take a long time for the new capabilities of libjpeg to make their way into camera firmware (if ever) due to memory and processing speed requirements of manipulating thàt much more data with a small processor and limited memoryspace in the camera hardware.

My personal opinion? Great if you shoot raw, convert to jpeg using the new capabilities with the intention to print. No huge impact when shooting/processing for web or OOC straight jpegs.

 

[Glen Barrington]

Well, most printers convert to 8 bit before printing anyway. I suspect the most predictable course of action in regards to color management is to handle that conversion yourself so you at least know how it is supposed to look in an 8 bit conversion. So it isn't clear to me how this new format would be a benefit to printing.

--
I still like soup. . .
Fine Art Flyover Country:
http://www.flickr.com/groups/2505841@N22

 

[newmikey]

Useful and informative post, but I think you might need to clarify your "personal opinion" about printing. I'm assuming you're talking about it being great for sending jpegs out to be COMMERCIALLY printed. For printing on a personal inkjet printer, you'd still be better (theoretically at least) working with 16-bit files and keeping them there throughout the whole workflow. Agreed?

Printing 16-bit TIFF both commercially as well as at home would trump even the newest 12-bit implementation of course. For the lazy (like me f.i.) who start out with raw and convert to jpeg for web-use as well as high-street printing (and sometimes online professional printservices which limit upload size) the 12-bit jpeg would be a great middle ground however...

 

[canonagain123]

How is that relevant? It does not disprove that CR2 is compressed with lossless JPEG, since TIFF is a container format which can be used for several different compressed and non-compressed image formats.

Sites which write "RAW" in capital letters, as if it's an acronym, annoy me immensely.

Me too, but how is that relevant?

there is no 14-bit JFIF standard, therefore the 14-bit data cannot be stored as compressed, uncompressed, encoded, unencoded, encrypted or unencrypted JPG anything. Was that clear enough for you, or do you need it phrased differently?

I'm saying 14-bit, because RAW images out of most cameras are 14-bit, but there are also 16-bit raw images, 24-bit, 32-bit, 48-bit and 64-bit raw images. There is no way to store any of those in an 8-bit container. The 12-bit JPG standard is not widely supported, and is not used outside medical imaging where the 12 bits are used to represent different shades of gray.

Would there be as many as 50 shades of gray?