The SRW format

Been hacking my way through the SRW file to make a raw averager (for multiple exposures). The SRW is actually is very much like a tiff file with exif information. The raw data is just packed 12-bit numbers with no compression (I actually appreciate that as it makes it much easier to work with). Its all big edian too. The most surprising thing though is that in addition to a thumbnail and preview the raw file contains an entire full resolution jpg. So when you choose RAW you're actually getting JPG + RAW in the same file. If you choose RAW + JPG your'e actually getting the RAW + JPG + JPG (well I can't actually check the last one as my camera is in for repairs). Seems a bit of a waste of space but I imagine they did it so you could still review the picture on the camera while shooting RAW. The only other cameras that do that are the Sigma's.

Of course if you want to develop a whole stack of raws very quickly you could use this to your advantage and use an application like dcraw or exiftool to just extract the JPG and only go back to the RAW if there's something you need to change. I might even write something like this myself if I have the time.

phototransformations wrote:

The DNG converter creates its own embedded JPEG, and you can specify the approximate size. It seems to discard the Samsung JPEG, rather than resize it, as the embedded JPEG (when extracted) more resembles a default ACR conversion of the SRW, saved at "Normal" compression, even if the original SRW file contained an embedded "Superfine" JPEG.

A program to extract JPEGs from RAW files already exists and works with the SRW files. It's called "Instant JPEG from RAW" and works fairly well, though the UI is a little irritating.

What Zdman wants to do is write his own program, probably in C or similar which is apparantly trivial and makes it possible to do various things on a PC to a very simple file, then save as tiff or jpg (e.g. using libtiff/libjpeg on unix, the program you mentioned ). That would interest me, when I have the time I'll look into it as well, as I wanted to do some numerical manipulation of lamp beamshots...

The program you mentioned is only available for windoze and mac, and for what seems to be trivial program, I'd rather make my own so I can incorporate into other programs of mine without restrictions. It seems Zdman wants to do a similar thing.

What Zdman wants to do is write his own program, probably in C or similar which is apparantly trivial and makes it possible to do various things on a PC to a very simple file, then save as tiff or jpg (e.g. using libtiff/libjpeg on unix, the program you mentioned ). That would interest me, when I have the time I'll look into it as well, as I wanted to do some numerical manipulation of lamp beamshots...

I've been also playing around, and it is not that trivial. The SRWs are not 100% tiff compliant. you cannot open them with libtiff, because there are some missing mandatory tags. Currently I have my srw/tiff parser half done, when I finish it, I'll make it public.

The program you mentioned is only available for windoze and mac, and for what seems to be trivial program, I'd rather make my own so I can incorporate into other programs of mine without restrictions. It seems Zdman wants to do a similar thing.

Maybe we can work together? (I'm working under linux).

Cheers,

Jose

Yeah not standard Tiff but I read this before I started looking so was prepared

http://metadatamadness.blogspot.com/2010/03/stupid-samsung-srw.html

and the March entry which I can't link due to the words in the title.

I only really want access to the raw data and the start/length of the various emedded images so the non-standard tags are not an issue. I'm making a small utuliity to avaerage raw files (all the available ones work on developed images). It will be in C and won't be more than a few pages of code (unless I decide to make a GUI)

Currently I have a small TIFF parser that goes trough the file, printing the IFDs. currently it works, but I have to polish it a bit. a bit similar to what tiffdump does, but with more information.
If you want it, just let me know.

Cheers,

Jose

I shoot RAW +SF jpeg pretty much all the time. One thing I've noticed is that RAW files converted to jpeg are 240 dpi. The jpegs from the camera are only 72 dpi. Is this normal for all cameras? 72 dpi seems awfully low if you want to do printing. Am I wasting my time and card space shooting SF? Will the 72 dpi actually effect printing if I want to print say a 16x20?

crsantin wrote:

I shoot RAW +SF jpeg pretty much all the time. One thing I've noticed is that RAW files converted to jpeg are 240 dpi. The jpegs from the camera are only 72 dpi. Is this normal for all cameras? 72 dpi seems awfully low if you want to do printing. Am I wasting my time and card space shooting SF? Will the 72 dpi actually effect printing if I want to print say a 16x20?

In this case dpi is just a note defining the default enlargement size, but you can print or display at whatever size you want. The resolution, which affects the maximum enlargement size, is the same between RAW and JPEG.

Ah, sorry for the noise, I misinterpreted a post on the imagemagick forum and thought dcraw was able to extract the embedded jpg. Post deleted.

Hi,
I just found this: http://www.libraw.org

It looks quite promising, and way less work. My plan is to write a small utility tat converts the SRW files into DNG under Linux, so I can recover the wasted space (among other plans).

Cheers,

Jose

Turns out it's pretty easy to write a python script to extract both jpegs: just mmap the file, then look for '\xff\xd8' followed by '\xff\xd9'. The portion of the file between those two identifiers, including them, is a jpeg.

Quick working code attached as a screenshot (no idea how to format code here on dpr), you can grab the script here

http://qix.it/site_media/samsung_srw_extract.py

jgardia wrote:

What Zdman wants to do is write his own program, probably in C or similar which is apparantly trivial and makes it possible to do various things on a PC to a very simple file, then save as tiff or jpg (e.g. using libtiff/libjpeg on unix, the program you mentioned ). That would interest me, when I have the time I'll look into it as well, as I wanted to do some numerical manipulation of lamp beamshots...

I've been also playing around, and it is not that trivial. The SRWs are not 100% tiff compliant. you cannot open them with libtiff, because there are some missing mandatory tags. Currently I have my srw/tiff parser half done, when I finish it, I'll make it public.

The program you mentioned is only available for windoze and mac, and for what seems to be trivial program, I'd rather make my own so I can incorporate into other programs of mine without restrictions. It seems Zdman wants to do a similar thing.

Maybe we can work together? (I'm working under linux).

Cheers,

Jose

Keep it up guys, it is beyond me, but I might comment that most camera manufacturers seem to want always to re-invent the wheel when it comes to raw file creation. Seems from your comments that there is no great magic in raw file composition just a bunch of in-house firmware crunchers doing their own thing regardless of what is already there in public domain. Or are they just obsessed with patenting proprietary raw software algorythms?

Maybe they just wish to provide a challenge for others to make the raw files generally available in more transparent fashion across products? Does anyone have any idea how many raw file formats now exist. How many last week will do, tomorrow there will be even more.

Turns out exiftool is able to extract the full size jpg and attach exif data from the raw file to it

exiftool -if '$jpgfromraw' -b -jpgfromraw -w %d%f_%ue.jpg -execute -if '$previewimage' -b -previewimage -w %d%f_%ue.jpg -execute -tagsfromfile @ -srcfile %d%f_%ue.jpg -overwrite_original -common_args --ext jpg data

where data is the folder containing srw files.

The number of jpg in the samsung raw files is not two but three: a very small thumbnail for image navigation, a scaled down preview, and the full jpg. Exiftool only extracts the full size jpeg.

ludoo wrote:

Turns out exiftool is able to extract the full size jpg and attach exif data from the raw file to it

exiftool -if '$jpgfromraw' -b -jpgfromraw -w %d%f_%ue.jpg -execute -if '$previewimage' -b -previewimage -w %d%f_%ue.jpg -execute -tagsfromfile @ -srcfile %d%f_%ue.jpg -overwrite_original -common_args --ext jpg data

where data is the folder containing srw files.

The number of jpg in the samsung raw files is not two but three: a very small thumbnail for image navigation, a scaled down preview, and the full jpg. Exiftool only extracts the full size jpeg.

Mentioned the preview, thumbnail as well as exiftool and dcraw being able to extract in my original post. As it happens I'm finished my raw averager and it works better than averaging the demosaiced images (there's a barely visible grid like pattern in those). I'll post the code if anyone is interested.

Zdman wrote:

Mentioned the preview, thumbnail as well as exiftool and dcraw being able to extract in my original post. As it happens I'm finished my raw averager and it works better than averaging the demosaiced images (there's a barely visible grid like pattern in those). I'll post the code if anyone is interested.

I was too excited to read about the embedded full res jpeg and I miss the fine details in your post.

Zdman wrote:

ludoo wrote:

Turns out exiftool is able to extract the full size jpg and attach exif data from the raw file to it

exiftool -if '$jpgfromraw' -b -jpgfromraw -w %d%f_%ue.jpg -execute -if '$previewimage' -b -previewimage -w %d%f_%ue.jpg -execute -tagsfromfile @ -srcfile %d%f_%ue.jpg -overwrite_original -common_args --ext jpg data

where data is the folder containing srw files.

The number of jpg in the samsung raw files is not two but three: a very small thumbnail for image navigation, a scaled down preview, and the full jpg. Exiftool only extracts the full size jpeg.

Mentioned the preview, thumbnail as well as exiftool and dcraw being able to extract in my original post. As it happens I'm finished my raw averager and it works better than averaging the demosaiced images (there's a barely visible grid like pattern in those). I'll post the code if anyone is interested.

Yes please, I'd like to see the code, if you don't mind sharing.

Thanks,
Peter

no code formating so you'll have to do a copy paste to a compiler. Essentially a command line tool so just type the names of the files you want to average after the command line and it'll output a SAM_AVERG.RAW. Its hard coded to work with NX10 and NX100 Raws but if you find the raw size for the other camera's you can just update the len variable number. It'll work with as many files as you can fit to the command line but don't exceed 16 as you may overflow the shorts I've used in memory (you're welcome to change this).

1. include

2. include

int main (int argc,char argv[]){

FILE file_id=NULL;

unsigned int srw_start=0;

FILE file_w;
unsigned char b1=0;
unsigned char b2=0;
unsigned char b3=0;

unsigned short s1=0;
unsigned short s2=0;
unsigned short sm=0;

unsigned int len=22028832;
unsigned int s_len=len*4/3;
unsigned int short_count=0;
unsigned int num_files=argc-1;

unsigned char mem_b; read buffer
unsigned char mem_b2; write buffer
unsigned char mem_b3; header buffer
unsigned short mem_s; pointer for short memory

mem_b=(unsigned char*)malloc(len); allocate image to be loaded

mem_b2=(unsigned char*)calloc(len*2,1); allocate short memory and set it to all zero
mem_s=(unsigned short*)mem_b2; cast it as shorts

for (int n=1;n
if(file_id!=NULL){
fclose(file_id);
}
file_id=fopen(argv[n],"rb");
if (file_id==NULL){
printf("could not open file to read \n");
printf(argv[n]);
return 0;
}else{
printf(argv[n]);
printf("\n");
}

fseek(file_id,0,SEEK_END);
srw_start=ftell(file_id);
srw_start=srw_start-len;

fseek(file_id,srw_start,SEEK_SET);
fread(mem_b,1,len,file_id);

short_count=0;

for (unsigned int i=0;i
b1=mem_b[i];
b2=mem_b[i+1];
b3=mem_b[i+2];

s1=b1;
s1=s1

sm=b2> > 4; shift the next byte 4 bits to the left so you're only left with the bottom 4

s1=s1+sm; add them together and you have your first short

s2=b3; 3 byte is the lower 8 bits of the next short

sm=b2&0x0F; blanks out the top 4 bits of the middle byte (you could also try shifting it 4 left)
sm=sm

s2=s2+sm; add the beginning and end together

mem_s[short_count]=mem_s[short_count]+s1; write the first short to memory
mem_s[short_count+1]=mem_s[short_count+1]+s2; write the second short to memory
short_count=short_count+2;

}

}

average

for(unsigned int n=0;n
mem_s[n]=mem_s[n] num_files;
}

packing and endian convert. The intel stors the most signficant byte last while the file wants it first so bytes have to be moved around

short_count=0;

for (unsigned int i=0;i

s1=mem_s[short_count]; I've done this in steps to keep it simple to understand you can do it on three lines if you wish

s1=s1> > 4; shift the the top 8 bits of the 12 bit number to the bottom (one byte)

b1=s1; fist byte now equal to first 8 bits

s2=mem_s[short_count]; still working off the first 12 bits now we need to put the last 4 bits in the second byte
s2=s2&0x000F; clears the top 12 bits
s2=s2

sm=mem_s[short_count+1]&0x0FFF; now we need the first four bits of the next 12bit number so blank out the top 4 just in case

sm=sm> > 8; shift the remaining bits down 8 and we should have the first 4 bits of the second 12 bits at the bottom of the short

s2=s2+sm; add the top and bottom bits together

b2=s2; put it in the second byte

s2=mem_s[short_count+1]; now we need the last 8 bits of the second short in the last byte

s2=s2&0x00FF; clear everything except the last 8 bits

b3=s2; put the last 8 bits in the last byte

mem_b[i]=b1; write bytes to memory
mem_b[i+1]=b2;
mem_b[i+2]=b3;

short_count=short_count+2;

}

free(mem_b2);

mem_b3=(unsigned char*)malloc(srw_start);

fseek(file_id,0,SEEK_SET);

fread(mem_b3,1,srw_start,file_id);

fclose(file_id);

file_w=fopen("SAM_AVER.SRW","wb");

fwrite(mem_b3,1,srw_start,file_w);
fwrite(mem_b,1,len,file_w);

fclose(file_w);

free(mem_b);
free(mem_b3);

}

Thanks for this code. Not tried it yet, but I will do soon. I'm interested in the EX1 raw files, so will presumably have to fix the figures as stated.

Thanks again,
Peter