High volume location storage

We are looking for an intelligent solution to the storage problems we think we'd face on location after switching over to digital capture from film.

At the high end, we finish a day with around 200 rolls of film - a mix of 35 and 120. If we were to use digital capture instead of film, it seems as though we'd be looking to store somewhere in the neighborhood of 50 - 80 GB of files for the day. In some cases a generator or motorhome could provide power for laptops or external firewire drives, but in other cases, we'd have to be self-sufficient and rely on batteries for cameras, laptops and storage media.

We'd also like to have some kind of redundancy in the storage (CD's and DVD's are too slow to burn I think) so that one or two hard drives crashing won't ruin the day. It seems like RAID or mirror drives would do the trick there.

I think I've read almost every post on storage, media and portable devices, but I'm wondering if there is someone out there that's come up with a better solution than we have: One, having a generator, a couple of laptops, and a stack of external firewire drives on the days that permit it; and two, having a enough portable devices (MindStor, etc.) and loads of batteries on the days we are truly in the woods, on a boat, or just simply out of our minds.

We appreciate all the information that this and other groups have made available, and hopefully in the near future we'll have learned enough about the process to contribute.
Thanks!

The following articles from Rob Galbraith's site on the digital equipment used to shoot the Eco-Challenge may be helpful.

http://www.robgalbraith.com/diginews/2001-10/2001_10_18_gear.html

http://www.robgalbraith.com/ ... ... 2001-10/2001_10_29_eco_gallery.html

Hi David

We would shoot on location quite a bit and would regularly go through 10+ GB per day.
50 to 80 GB is a lot.

Remember with digital you won't have to bracket and you can stop shooting once you see you have the result. This may mean less shooting, for your sake I hope so.
Anyway, a second internal hard drive is probably your best bet.

We have two G3 PowerBook's with 30Gb hard drives, and a 30Gb second hot swappable hard drive to copy too.

At the end of each day we use a cross over cable to transfer the images over onto our G4 desk tops - which have 2x 80Gb hard drives. I found the firewire cable faster at transferring; however, some the the files/ images were corrupt.... not good news. The cross over cable, so far, is 100% reliable... it will take about 30 to 40 minutes to transfer 10Gb, so in your case the transfer could take all night. One good thing about computers is that they don't need to sleep.

We usually bring one of the G4's with us when on location for more than a few days, transferring, processing and burning CD's at the end of each day.

It will take you DAYS sorting thur 80Gb of images, as I said before, that's a lot of images.

The most we have every shot is about 17Gb with the medium format system, that's about 500 x 34.6Mb raw files.

The 35mm systems raw files are only 7.7Mb so even a thousand images only equals 8Gb.

We try not to delete anything during the shooting day, as this has proved very dangerous.

The Apple PowerBook battery will run for about 3 to 4 hours, so 3 or 4 batteries should see you safely through any day.

We have a "cigar lighter" charger for the PowerBook's as well as a 300 watts AC/DC converter which can be either plugged into the cigar lighter socket or connected to a small motor cycle battery... very handy for location work... light, rechargeable, and cheap to buy.

You did stay what it was you were shooting or how good the quality had to be, the reason I ask is because shooting in Jpeg format may be your answer, less than 3Mb per image.

Ashley.

http://www.ashleymorrison.com

Ashley,

Thanks for the input. Our estimates of how much storage space we'd need may be way off, although I do know that it sometimes takes days to go through the film from each day we shoot this kind of stuff, so I assume it would be similar when we switch to digital. We'll find out next week when we start.

Have you found that files become corrupted when you transfer via firewire to both external hard drives and to CPUs (internal hard drives)?

Unfortunately from a storage point of view, we do have to stick to the raw files. A lot of the images would need the extra range that the raw file provides.

I think that since most of our work is done outside and AC power is not available, the best bet is a small generator, laptops, and external firewire drives. We'd transfer the data at night and back up to CD's or DVD's.

Do you use the internal CD writers in the G4's or do you use a faster external CD writer? It seems that the 40x writers would save considerable time over the internal (8x?) writers in the Macs.

Thanks for the information - next week, I'm sure we'll have our own lessons to share.

Hi David,

DavidFoster said:

I think that since most of our work is done outside and AC power is
not available, the best bet is a small generator...

Click to expand...

I suspect that you might be happier with a deep cycle battery and a pure sine wave inverter. That way you can avoid the noise/fumes/refueling problems of the generator and have much 'cleaner' power for your more sensitive electronics.

--
bob
http://www.pbase.com/bobtrips
New Gallery - Nat (Spirit) Festival in Myanmar
pictures from Thailand, Myanmar(Burma), and Nepal

DavidFoster said:

I think that since most of our work is done outside and AC power is
not available, the best bet is a small generator, laptops, and
external firewire drives. We'd transfer the data at night and back
up to CD's or DVD's.

Click to expand...

I have no experience the levels of photography you're considering... but a computation solution seems to be key. If you're thinking about using/moving a generator you might as well use a RAID storage solution... This would allow for large amounts of storage (hundreds of GBs) as well as redundancy (as to a single drive malfunction... not against theft or natural desires). Firewire RAID devices are available (require power to run though) and a G4 PowerBook is more than capable of handling any of the computing needs on site.

I currently use my G4 PowerBook onsite to transfer/store images when my CF cards fill up (while I'm not using a fancy DSLR, I only have a 256MB card and it fills up after 144 images). I use a PC Card adaptor, but you might want a FW card reader for speed purposes, to transfer the images to my 60GB internal HD... later when I get home I transfer them to my 4x120GB FW drives (in your case RAIDed for backup).

If you planned on doing something like this it will cost some $$$, but considering the alternative of multiple FW drives, laptops, batteries, CDs, etc... it might be what you're looking for.

-Todd...

Note: If you were not too remote (had close proximity to a car/van) you could setup a portable storage system, which when done with the shoot could be transfered into a studio for editing/etc...

I would look at yanking the internal CDRWs and replacing them with internal DVD-RWs. The current crop of DVD burners will burn 4.7gb in about 15 minutes. Sony just released a great DVD burner that burns both the "-" and "+" format DVDs and costs $350.

Firewire drives are just cheap IDE drives in external boxes, so they will never exceed the speed of an internal drive. Save a few bucks and an extra piece of equipment. It will also not require external power.

DavidFoster said:

Do you use the internal CD writers in the G4's or do you use a
faster external CD writer? It seems that the 40x writers would
save considerable time over the internal (8x?) writers in the Macs.

Click to expand...

You look into the Digital Wallets or other like equipment.

I carry a 3gig DW as I'm just a hobbist photog, but they do make larger capacity DW's.

Battery operated and USB connectivity.

If you do get a DW, I suggest you charge the batteries before each event.

--
Vance.

http://users.ev1.net/~txcowboy
http://pub57.ezboard.com/bthedigitaldinguscommunity

DavidFoster said:

We are looking for an intelligent solution to the storage problems
we think we'd face on location after switching over to digital
capture from film.

At the high end, we finish a day with around 200 rolls of film - a
mix of 35 and 120. If we were to use digital capture instead of
film, it seems as though we'd be looking to store somewhere in the
neighborhood of 50 - 80 GB of files for the day.

Click to expand...

That is a LOT of data to store on a daily basis, you're going to need a custom solution and deep pockets if you intend to keep much more than a days worth of images on hard disk with any sort of reliability. Are you really sure you'll be generating that much data daily?

DavidFoster said:

In some cases a
generator or motorhome could provide power for laptops or external
firewire drives, but in other cases, we'd have to be
self-sufficient and rely on batteries for cameras, laptops and
storage media.

We'd also like to have some kind of redundancy in the storage (CD's
and DVD's are too slow to burn I think) so that one or two hard
drives crashing won't ruin the day. It seems like RAID or mirror
drives would do the trick there.

Click to expand...

You're correct about RAID being the way to go for reliability (specifically RAID 5 is what you're looking for, using an array of 3 disks with data redundantly striped across all three). IDE drives won't cut the mustard at this level, you'll be looking to SCSI arrays which unfortunately means a significant increase in cost.

There are only three possible solutions I can come up with, none of which solve all the problems you face.

1 : Base station and RAID array.

A custom built PC & raid array could be constructed in a relativly portable form (think 19" rack mount). Storage capacity ranges from around 30Gb all the way up to 2Tb and depends largely on how much you're willing to pay for. Bear in mind drives need to be purchased in lots of 3 for raid-5 operation so a 180Gb array actually contains 3x180Gb drives. You can house 4 such arrays per housing and up to 2 housings can be used together. I'd be inclined to recommend 3 arrays of around 70-80Gb each, giving 3 days online storage. A tape drive would be required for archive storage, DVD has a maximum capacity of 4Gb per disk for the writable variants currently available. This is obviously a tethered solution, suitable only where there's a source of power but does provide the best option in terms of data security & speed (data can be archived to tape unattended overnight). Suitable 19" housings are available from Pinnacle (and others), SCSI cards from Adaptec and drives from various sources (IBM, Segate etc.)

2 : Laptops with DVD burner & large disk capacity. Much more portable but the maximum storage you're likely to get is 80Gb and there's no redundancy so you'd be looking at burning several DVD's every night. Ideally you'd want a laptop with provision for 2 internal disks (I think compaq may have such a model but I'm not certain), store th data on one and the OS on the other for ease of use.

3 : Portable storage. Best for portability and you can go up to around 80Gb per unit but these have no data redundancy so you'd be at the mercy of disk failure (may or may not be a significant risk, depending on the environment). Upside is these are relativly inexpensive so you could use 1 per day's shooting and worry about data transfer back at base.

Depending on the duration of your trips & the arrangements you make a combination of these solutions (perhaps 1 & 3 or 2 & 3) may well be most appropriate.

All in, it's a difficult challenge to store that much data daily even in a well-controlled office environment, doing so reliably in the field is considerably harder.

HTH,

Jim.
(full-time IT consultant for more years than I like to remember!)

BobTrips said:

Member said:

I think that since most of our work is done outside and AC power is
not available, the best bet is a small generator...

Click to expand...

I suspect that you might be happier with a deep cycle battery and a
pure sine wave inverter. That way you can avoid the
noise/fumes/refueling problems of the generator and have much
'cleaner' power for your more sensitive electronics.

Click to expand...

You will want to run off a battery to avoid the possiblity of having a generator shut down in the middle of data transfer - an uninteruptable power supply is really required. You want enough power in the batteries to go long enough to gas up the generator with time to spare.

A load of 120 watts is 10 amps at 12 volts so an 80 amp-hour battery would go to zero in 8 hours, neglecting inverter loss and the fact the inverter will cut out before the whole 80 amp-hours were extracted. A generator and gas are lighter than the batteries you would need for a couple of days.