Looking for advise to manage my RAW library, wanna share you tricks ?

Hi All,

Basically I am not really satisfied of the way I manage my RAW files...

I am however happy with the rest of my workflow, the working pictures, the final exports, etc. But I don't like the workflow around my RAWs.

What I do so far:

• Dump all my raws from the compact flash (I use D700) to a folder called RAW.
• I do create a Subfolder, and if I have time, I'll create more than one and try to group the pictures taken on that card. Like: Central Park Afternoon, Lower Est side night, etc.
• Then I take Adobe Bridge to preview my images, make a selection of what I like and what to keep.
• Then I copy that selection to another folder called Edit where I usually respect the same subfolder structure
• Then from bridge in the new location, I will import those in Lightroom or Dxo (depending how I want to process them)
• And when I need too I'll do a second pass in Photoshop
• When I happy with the result I export my JPG to an export folder (same subdirs)
• Finally I add the JPG to my iphoto album which I then use to upload to Flickr or others...

So all in all, I got a pretty much standard approach. But what I really don't like is how my RAW are stored, it's messy, it does not take into accounts the dates, etc.

WHen I was using windows, I really like how picassa was doing things, taking each folder and "guess" the average date, you could then find them back on some sort of timeline quite easily.

I am looking of some way to do something similar I guess, but I actually hate sorting.... I would love some way to group the picture by the time they were taken ? I don't know something like all the pictures taken whithin 30 min should belong to the same group ? Sort the groups by dates, maybe add some critiria keyword to facilitate the search later ? Like camera model ? Lens used, etc. ?

Any idea, suggestion how to achieve that sort of things ?

Alex
--
http://www.flickr.com/photos/alexandre_ponsin/

I used to dump all my RAW files into a RAW folder but eventually (didn't take long) it became hard to find anything.

I got Downloader Pro to do the downloading now. It fires up when I put a card into the card reader and I have it set up to ask for a job code at startup. The job code is just a text string used to describe the subject, e.g. "Suzies Birthday Party". The program then displays thumbnails of the photos. I select the ones appropriate to the job code, and download them. The program stores them in a folder named by the job code (generating a new folder if one doesn't exist). That folder is a subfolder of a folder with the year as the name. The individual files are named by the job code, the date the picture was taken, and a 3 digit index number (I've never taken more than 500 pictures of any one thing, but the program could use a 4 digit number). Thus the files are named "Suzies Birthday Party 091106 001.NEF" etc. (I always write the date as YYMMDD because it sorts properly). The files are located in \user\me\2009\Suzies Birthday Party\RAW. If you shoot jpg, the program will store them in ...\JPG. If you shoot both, the program will store the appropriate file in the appropriate folder. The nice thing about this program is that the only thing I have to give it is the job code and the rest is automatic (based on the way I have it set up). The other nice thing is that the files have a name that gives you an idea what subject you took the photo of, rather than DSC_6201.NEF, which means nothing to anyone.

After I download all the pictures with a given job code, I can change the job code and highlight pictures appropriate to the new one and download them, again all automatically to the appropriate folders.

I now have all my RAW files stored in folders by year and a descriptive name. When I run them through Lightroom I can save the pictures in the folder just above the RAW folder. Sometimes I generate an Archive folder for the unimportant pictures that went through Lightroom. When I run them through Lightroom I try to put appropriate tags on them.

So now I have two ways to find a file. By year and descriptive text, and by looking for tags in Lightroom. Actually Windows will look at the tags too, so that's another way to find things (as long as your tags are reasonably descriptive).

PS: the file name and path are determined by setting up the options in Downloader Pro. There are lots of things you can use, including camera name, serial number (Canon only), lens, ISO, shutter speed, aperture, and other stuff. I only use one camera at a time, so job code,date,index# is sufficient for me. A wedding shooter with two or more shooters/cameras might want to add other things to separate the shots. I believe that if two files somehow end up with the same name, Download Pro will not let you download the second one until you change the name somehow.

My only problem is that I use an index number that starts at 001 for all downloads. If I have more than one card it will try to reset the index number for the second card the way I have it set up. I could set it up to be sequential, but then the index numbers for the next shoot won't start at 001 unless I reset them and I'm trying to do everything automatically so I don't have to enter anything except the job code. There's nothing inherently wrong with having the index numbers start at 527, but it's just not aesthetically pleasing. I expect there's a good way to do it, but it's such an infrequent problem for me I haven't worried about it much. Most of my shoots fit on one 4 Gb card.

It's really hard to answer these kinds of questions, because different photographers have different needs.

Over the last seven years, I've evolved a system where I organize everything into folders by year, month, day, and then subject by day. So if I need to find, say, a Lenny Kravitz concert I shot on June 26 in 2005, I simply go to the 2005 disc, open the June folder, and then find the Lenny Kravitz file.

Inside the file, I have subdirectories for RAW selects, RAW nonselects, Final Hi Res images, and sized for web.

Ah, you ask -- what if you can't remember the date of the session? That's where keywording comes in. Using Mac Spotlight, I simply search for "Lenny Kravitz" and it find all the images I've ever taken of him. Spotlight is so fast and easy to use, I just don't understand why more power is needed.

I find this far superior to using an asset management system. The thing I hate about all the asset management systems I've tried, and that includes long-term use of Lightroom, Aperture, and Extensis, is that you are essentially running a data base on top of a date base. It adds another layer of complexity -- another set of files that can be lost or damaged, with severe consequences. After years of trying, I am completely unconvinced that the extra level of work and hassle involved with a DAM produces meaningful rewards.

After messing around with asset management systems for several years, I have stopped using them, and find the simple organization by date, combined with robust keywording, to provide the best of all possible worlds.

I like to use Photo Mechanic for my initial editing and keywording, because it is screamingly fast, and doesn't bog down creating thumbnails and previews like many other programs do when you have several thousand images in the file, such as when processing a wedding.

Once I have made my selects and put the selected images into a RAW select folder, I close PhotoMechanic, and use Adobe Bridge and Camera RAW to do a first pass editing. This is where I fix exposure, color balance, cropping, straightening, etc. Then I do a batch conversion into jpg, and store those images in the Final Hi Res file. Finally, I go through the Final Hi Res images, and select images for additional treatment -- retouching, filters, etc. in Photoshop.

This workflow has served me well for everything from commercial photography to newspapers, to concert work, glamour, headshots, and weddings. My library has hundreds of thousands of images, and spans over a dozen 1, 1.5, and 2TB drives -- and that doesn't include the backups.

Hope this helps,
Paul
http://upstatephotographers.com
http://albanyheadshots.com

My way of handeling this problem was by buying and using lightroom for everything.
I find that it is a lovely app.

However you run into the problem that you really have to tag and devide in collections your photo's if you want to find them again quickly. Well at least if you have a big library. (mine is at 30K+ at the moment)

However it works for me and I can recommend you give it a serious spin.

Paul Grupp said:

It's really hard to answer these kinds of questions, because different photographers have different needs.

Click to expand...

Thanks for sharing, Paul. Always interesting to get to know how folks who have been in business for a long time are managing their workflow.

Paul Grupp said:

Once I have made my selects and put the selected images into a RAW select folder, I close PhotoMechanic, and use Adobe Bridge and Camera RAW to do a first pass editing. This is where I fix exposure, color balance, cropping, straightening, etc. Then I do a batch conversion into jpg, and store those images in the Final Hi Res file. Finally, I go through the Final Hi Res images, and select images for additional treatment -- retouching, filters, etc. in Photoshop.

Click to expand...

What I don't understand is that step "batch conversion to jgp" -- why would you reduce your files to 8 bit before your final retouching work in PS? Or did I get you wrong?
--
regards, eric

Alexandre Ponsin said:

I am looking of some way to do something similar I guess, but I actually hate sorting.... I would love some way to group the picture by the time they were taken ? I don't know something like all the pictures taken whithin 30 min should belong to the same group ? Sort the groups by dates, maybe add some critiria keyword to facilitate the search later ? Like camera model ? Lens used, etc. ?

Any idea, suggestion how to achieve that sort of things ?

Alex
--
http://www.flickr.com/photos/alexandre_ponsin/

Click to expand...

Far less than perfect but...

Background: I currently use photoshop and bridge, external USB drives, and Zenfolio...

After any session, I create a folder on my WORK drive with a filename reflecting the date and subject matter (yymmdd+subject). I then copy the contents of all cards for the session into the folder and rename them using the folder name with a sequence number.

I then create a second folder within the folder with the same name plus the phrase "on Zen". Once this is done, I copy the session to general subject folders two other USB drives.

I then process sessions with each folder using PS & bridge. After each group of 20-40 shots, I load the (hi-res) jpgs to Zenfolio and move the NEF/XMP/JPG for this group to the 'on Zen' folder for that session.

Once the entire session is processed, I replace the unprocessed backup copies (on the other two USB drives) with the 'yymmddSUBJECT on Zen' folder and remove it/them from the work drive. I also blow DVDs.

This leaves me with two live copies of the full session (NEF/XMP/JPG) on USB drive, hi-res copies online at Zenfolio, and the DVDs. With Zenfolio's capabilities, I can create public and private viewing groups built from the single uploaded copy there.

I have given thought to other programs (such as Lightroom) but haven't been able to determine how they might adapt to the manner in which I keep my working and backup copies (across drive). Another factor affecting my workflow (and choice of tools) is my process of making final composition and cropping decisions outside the camera based on the subject. (I rarely end up with a standard format which limits how much I can do in batch mode.)

--
Ric

Because I do all the basic color correction, exposure compensation, etc. in RAW. Once I go to Photoshop to do things like zit removal, or to run filters like Nik SilverEFX Pro, ColorFX Pro, or Vivezia, you have already left the RAW domain anyway.

It's a pragmatic thing. 99 times out of 100, after color and exposure correction, there isn't much value in working on the RAW image -- and to use some tools, you can't work on the RAW image anyway. And in that 1 out of 100 times where you discover there is something wrong with the file that requires a tweak in the RAW converter, you're better off going back to the RAW file for that one file, than going through the slow process of converting hundreds of RAW files to jpg individually.

Paul Grupp said:

It's a pragmatic thing. 99 times out of 100, after color and exposure correction, there isn't much value in working on the RAW image -- and to use some tools, you can't work on the RAW image anyway.

Click to expand...

I think I understand your logic. But I'm wondering why you don't keep the 16 bit when leaving raw domain by converting to 16-bit Tiff instead of 8-bit Jpg files -- those 8 bit plus provided by 16-bit Tif format would give you more leeway for retouching like finer tonal gradations, at least, that's what I'm thinking.
--
regards, eric

The thing is -- if I put a .45ACP pistol to your head, and placed a stack of prints before you -- some edited in 16 bit, some in 8 bit, and told you I would pull the trigger the second print you guessed wrong -- how long do you think you would live?

I'm guessing not more than a few seconds. Now if YOU -- an advanced amateur can't see an actual tangible benefit by examining prints closely, what chance do you think my far more naive and uneducated clients have?

In the business world, we all make decisions about what to include in our processes, and what to leave out. One things for sure -- if every business did every process to perfection, nobody could afford anything in this world. Similarly, if you cut too many corners, you produce junk.

The trick is in knowing what tradeoffs to make.

Just tagging - interesting toppic

if you have the a jpeg of an image from a job and you want to "find" the raw, you could search for the number..if you keep resetting you will have lots of hits for frame 001

if you allow consecutive numbering, the same "find file" will yield many fewer hits..

Paul Grupp said:

In the business world, we all make decisions about what to include in our processes, and what to leave out. One things for sure -- if every business did every process to perfection, nobody could afford anything in this world. Similarly, if you cut too many corners, you produce junk.

The trick is in knowing what tradeoffs to make.

Click to expand...

Thanks, I get your point. Obviously you've found a perfect mix for your needs. Myself, I'm using LR actually, but for some reasons you've described in your first post here I've mixed feelings about what you called 'adding another layer of complexity".
--
regards, eric

Thank you so much for describing your approach. The downloader pro thingy sounds like a neat little program. I am probably going to give it a try... Can it create folder by Year / Month instead of just Year ? I would like more granularity...

Do you think it is possible to rework all my existing collection through that program ? I have about 130 gig of RAW so far...

Anyway, thanks for sharing ! It's much appreciated !

Alex
--
http://www.flickr.com/photos/alexandre_ponsin/

Hi Paul,

I see your point about one Library on top of another, and I fully agree, which is why I am not managing my raw through Lightroom or other system... I find it to much of a hassle...

I am trying to come up with a good directory structure, to organize my RAW, and I also like the power of spotlight. But since I am a lazy guy I am looking for some way to improve that folder creation/renaming/tagging thing that I hate doing manually.

I am going to try that downloader pro program, and if I don't like it maybe I'll build my own...

I'll keep you posted anyway.

Alex
--
http://www.flickr.com/photos/alexandre_ponsin/

--
http://www.flickr.com/photos/alexandre_ponsin/

First, I should share my philosophy about data. All data files fit three categories - irreplaceable (raw file), hard to replace (edited file), easy to replace (drivers, commercial software). Plus, I've seen so many companies come and go, that I want the flexibility to move to any system (Mac or Windows). I don't mind performing a batch conversion in order to move to a new system.

I have my primary workstation (fast) and older PCs on a network with a robust HP Proliant server. In Lightroom, I import direct from the CF card to both the PC and Server (thus have 2 copies of irreplaceable files). Client shoots go into a Jobs folder, everything else into dated folders (year-month-day). Everything is given at least a single keyword. General shooting with lots of subjects get tagged with the keyword "No_Keyword", so I can search and tag them later (if I'm in a hurry).

I'll make my selections, minor edits and crops, then export into a "LR_Export" folder structure, named for the job, event, site or person I'll be sharing the photos with.

Anything that requires PhotoShop, gets exported from LR as a PSD file, ProPhoto color space, and reimported into LR. This way I can find any photo via LightRoom.

When my client work and non-client work started to get large enough, I went for two LR catalogs. I keep the two separate for a simple reason - Client work has a contract and possible license issues, while personal work doesn't.

Since Lightroom is based on the SQLite database engine, which is in the public domain. You can get database tools from lots of sources that allow you to peek into the database. Of course, you won't understand what most of it means, but you can extract keywords and filenames. If something happens to Lightroom or Adobe, I'll export to PSD, TIFF, JPG or DGN and decide what to do at that point. But Adobe seems fairly strong, and I can move to PC or Mac as I wish.

Oh, I forgot - I have a batch file that performs a backup to server, then shuts down the PC. The server backs itself up ( a long story) and e-mails the status to my iPhone. I find a server to be vastly superior to USB or Firewire drives. You can get a NAS (network attached storage) server for very little money these days and have all the files always online. And you don't have to worry about dropping a USB drive on the floor.

To sum it up: Server for second copy, with auto-backup. All files managed in Lightroom. Organized by client or non-client shoots. Other PCs can edit if primary PC were to die.

--
Ken Elliott
Equipment in profile.

So nobody knows about a Downloader Pro equivalent on macosx ?
--
http://www.flickr.com/photos/alexandre_ponsin/

Photo Mechanic is used by many, many professional photographers and news organizations. Robust, fast, feature-rich and inexpensive.

Sounds cool, I'll have a look. I am juste worried that Photo Mechanics does too much. Because the selecting, browsing part, etc. I like to do it with Bridge because of it's seamless integration with LR or PS.

Alex
--
http://www.flickr.com/photos/alexandre_ponsin/

Alexandre Ponsin said:

Sounds cool, I'll have a look. I am juste worried that Photo Mechanics does too much. Because the selecting, browsing part, etc.

Click to expand...

I believe you will be quite impressed with Photo Mechanic. I used it for a while, and I wasn't using Lightroom, I' be using Photo Mechanic. It is blazing fast, and I really liked it. I suggest downloading a trial version.

--
Ken Elliott
Equipment in profile.