Storage Solution for Large Photo Collection

Started 2 months ago | Discussions
BillyBobSenna
BillyBobSenna Senior Member • Posts: 2,049
Storage Solution for Large Photo Collection

Does anyone have a good storage recommendation for a large photo collection?

I have a higher end PC with a Lightroom catalog of about 700,000 photos spread across 5 internal hard drives. In addition, I have a Drobo 5C that serves as my backup for the 5 internal drives. I have no offsite backup.

I was thinking of moving my Lightroom catalog from the 5 internal drives to an external system and continuing to use the Drobo as a backup. I was hoping to have an external array device that appeared as 1 volume and was easier to expand as my collection grows.

Any recommendations?

-- hide signature --

Bill - Beverly Hills, MI
Motorsports Photography
www.billgulkerphotography.com

 BillyBobSenna's gear list:BillyBobSenna's gear list
Nikon D500 Nikon D850 Nikon Z6 Nikon AF Nikkor 50mm f/1.8D Nikon AF-S Nikkor 24-70mm f/2.8G ED +11 more
Richard James Regular Member • Posts: 197
Re: Storage Solution for Large Photo Collection

BillyBobSenna wrote:

I have a higher end PC with a Lightroom catalog of about 700,000 photos spread across 5 internal hard drives.

I would assume that you mean the images are on 5 drives, the catalog (database) itself is on one drive.

I was thinking of moving my Lightroom catalog from the 5 internal drives to an external system

Do you need all 5 drives online for immediate access? If not then I would suggest the current and last drive  (plus their backup) be online, the rest off-line.

Remember to keep at least 2 independent, and current, copies of each drive.

and continuing to use the Drobo as a backup

Personally I would keep drive pairs (master and backup) all on separate devices, not shared devices.

. I was hoping to have an external array device that appeared as 1 volume and was easier to expand as my collection grows.

If you do that you will have to go through your catalog and reset LR's drive pointers...folder by folder.

It is a pity that Adobe still uses drive letters to identify drives rather than volume lables. I have long ago exceeded the available drive letters so my catalog has multiple drives listed together, a PITA, but manageable ("duplicate" letters are well spaced in time so unlikely to be on-line at the same time).

Richard

BillyBobSenna
OP BillyBobSenna Senior Member • Posts: 2,049
Re: Storage Solution for Large Photo Collection

Richard James wrote:

BillyBobSenna wrote:

I have a higher end PC with a Lightroom catalog of about 700,000 photos spread across 5 internal hard drives.

I would assume that you mean the images are on 5 drives, the catalog (database) itself is on one drive.

Correct. I also  have a 1TB SSD as my system drive. The actual LR catalog is on the SSD.

I was thinking of moving my Lightroom catalog from the 5 internal drives to an external system

Do you need all 5 drives online for immediate access? If not then I would suggest the current and last drive (plus their backup) be online, the rest off-line.

Remember to keep at least 2 independent, and current, copies of each drive.

I would like to maintain online access to all my photos. All I have is 1 onsite backup on the Drobo. I am delinquent with creating a second offsite backup.

and continuing to use the Drobo as a backup

Personally I would keep drive pairs (master and backup) all on separate devices, not shared devices.

Not sure what you mean here. Not use a Drobo? Why? If 1 drive in the Drobo goes bad, all data is still ok.

. I was hoping to have an external array device that appeared as 1 volume and was easier to expand as my collection grows.

If you do that you will have to go through your catalog and reset LR's drive pointers...folder by folder.

It is a pity that Adobe still uses drive letters to identify drives rather than volume lables. I have long ago exceeded the available drive letters so my catalog has multiple drives listed together, a PITA, but manageable ("duplicate" letters are well spaced in time so unlikely to be on-line at the same time).

Richard

Thanks for the feedback Richard. Do you have any drive recommendations?

-- hide signature --

Bill - Beverly Hills, MI
Motorsports Photography
www.billgulkerphotography.com

 BillyBobSenna's gear list:BillyBobSenna's gear list
Nikon D500 Nikon D850 Nikon Z6 Nikon AF Nikkor 50mm f/1.8D Nikon AF-S Nikkor 24-70mm f/2.8G ED +11 more
ajscullard Contributing Member • Posts: 835
Re: Storage Solution for Large Photo Collection

BillyBobSenna wrote:

Does anyone have a good storage recommendation for a large photo collection?

I have a higher end PC with a Lightroom catalog of about 700,000 photos spread across 5 internal hard drives. In addition, I have a Drobo 5C that serves as my backup for the 5 internal drives. I have no offsite backup.

I was thinking of moving my Lightroom catalog from the 5 internal drives to an external system and continuing to use the Drobo as a backup. I was hoping to have an external array device that appeared as 1 volume and was easier to expand as my collection grows.

Any recommendations?

I would suggest doing a search here and in the retouching forum for "storage" and "archiving" - there have been several threads addressing this issue, mostly getting the same responses.

I use several internal drives designated as Primary and Backup to store all the actual files, a Synology NAS for secondary backup and several bare drives which I plug into a drive dock as required to provide off-site backup. I have always used Hitachi (HGST) drives and (touch wood) I have never had a drive failure in over ten years. I believe that Western Digital (WD) bought out HGST a few years ago, so if were buying new NAS internal drives I would probably look at WD.

I don't really use Lightroom, but I understand that it is best to keep the catalog on the same drive as your system, or at least on an internal drive simply for fastest performance. Backups of the catalog should be on separate drives.

Carey Brown
Carey Brown Senior Member • Posts: 1,940
700,000? How many giga-bytes?
1

BillyBobSenna wrote:

Does anyone have a good storage recommendation for a large photo collection?

I have a higher end PC with a Lightroom catalog of about 700,000 photos spread across 5 internal hard drives. In addition, I have a Drobo 5C that serves as my backup for the 5 internal drives. I have no offsite backup.

I was thinking of moving my Lightroom catalog from the 5 internal drives to an external system and continuing to use the Drobo as a backup. I was hoping to have an external array device that appeared as 1 volume and was easier to expand as my collection grows.

Any recommendations?

700,000 photos is a heck of a lot of photos? Are you a professional?

You might want to do some ruthless culling to get that to a more manageable size.

How many giga-bytes of storage does it take up?

For me, I always want my entire collections of original photos and video on a single drive. Western Digital has drives all the way up to 10TB. I can't imagine your collection taking that much.

A NAS can be 'a' backup but shouldn't be treated as 'the' backup (this assumes that all of your originals are on an internal drive). A power surge or fire could take out the entire unit. Additionally, if you have an 'oops' and accidentally delete a file, that deletion will probably be duplicated leaving you with no recoverable copy. It doesn't address the need for an off-site backup.

-- hide signature --
CAcreeks
CAcreeks Forum Pro • Posts: 13,754
Re: 700,000? How many giga-bytes?

Carey Brown wrote:

BillyBobSenna wrote:

Does anyone have a good storage recommendation for a large photo collection?

I have a higher end PC with a Lightroom catalog of about 700,000 photos spread across 5 internal hard drives. In addition, I have a Drobo 5C that serves as my backup for the 5 internal drives. I have no offsite backup.

I was thinking of moving my Lightroom catalog from the 5 internal drives to an external system and continuing to use the Drobo as a backup. I was hoping to have an external array device that appeared as 1 volume and was easier to expand as my collection grows.

Any recommendations?

700,000 photos is a heck of a lot of photos? Are you a professional?

You might want to do some ruthless culling to get that to a more manageable size.

How many giga-bytes of storage does it take up?

For me, I always want my entire collections of original photos and video on a single drive. Western Digital has drives all the way up to 10TB. I can't imagine your collection taking that much.

A NAS can be 'a' backup but shouldn't be treated as 'the' backup (this assumes that all of your originals are on an internal drive). A power surge or fire could take out the entire unit. Additionally, if you have an 'oops' and accidentally delete a file, that deletion will probably be duplicated leaving you with no recoverable copy. It doesn't address the need for an off-site backup.

Does your Windows version support RAID-10?

https://www.tomshardware.com/news/how-to-set-up-raid-windows-10,36783.html

Linux has LVM to present multiple drives as a single mount point.

With photos spread over 5 drives, it might be easier to buy external NAS and copy everything, or as Carey suggested, buy some 10TB drives.

Robert Zanatta Senior Member • Posts: 1,510
Re: Storage Solution for Large Photo Collection
1

Get a second Drobo?  Or another brand of DAS...

 Robert Zanatta's gear list:Robert Zanatta's gear list
Canon EOS 5D Mark IV
DerKeyser Regular Member • Posts: 439
Re: Storage Solution for Large Photo Collection
3

BillyBobSenna wrote:

Does anyone have a good storage recommendation for a large photo collection?

I have a higher end PC with a Lightroom catalog of about 700,000 photos spread across 5 internal hard drives. In addition, I have a Drobo 5C that serves as my backup for the 5 internal drives. I have no offsite backup.

I was thinking of moving my Lightroom catalog from the 5 internal drives to an external system and continuing to use the Drobo as a backup. I was hoping to have an external array device that appeared as 1 volume and was easier to expand as my collection grows.

Any recommendations?

Why complicate matters so much with drives all over the place?

Get a QNAP NAS as your primary storage. The list of advantages is just huge:

1: It does Raid 6, so it can handle two drive failures in a set (fx 6x8Tb drives = 32Tb Usable space).

2: It’s always on and allows access to your images from all devices including your phone, Ipad and so on.

3: It can run a nigthly backup job that Syncs (and encrypts) your images to a cloud service

4: It has a frontmount USB port and “one button backup” backup, that allows easy attachment of a USB disk for manual backups by just the push of a button.

5: It does internal snapshotting of its diskspace. So any disasters to data on the NAS (Ransomware fx), can be protected against and rolled back by a single snapshot restore.

My lightroom catalog is placed locally on SSD, my current working year is placed locally on HDD but has a SYNC job running that copies all images to the NAS automatically. All older images (25K+) are on the NAS only.

No more drives all over the place, a much much much much much better backup solution that runs completely automatic every night. On top of that there is the second manual backup copy to a USB drive I have.

Yes I know, it sounds complicated and “dangerous” with cloud sync. But it’s not. It’s simple to setup, WAY WAY better reliability, protection and backup, and all cloud data is encrypted (client side) so it’s just garbage to anyone but you.

I even wrote a little blog on the advantages of using a NAS in this scenario:

http://wolffmadsen.dk/articles/using-a-nas-to-secure-my-photo-workflow

-- hide signature --

Happy Nikon Shooter
See my articles and galleries at: https://wolffmadsen.dk

BillyBobSenna
OP BillyBobSenna Senior Member • Posts: 2,049
Re: Storage Solution for Large Photo Collection

Thank you Wolff. I will study your information. It looks very interesting.

-- hide signature --

Bill - Beverly Hills, MI
Motorsports Photography
www.billgulkerphotography.com

 BillyBobSenna's gear list:BillyBobSenna's gear list
Nikon D500 Nikon D850 Nikon Z6 Nikon AF Nikkor 50mm f/1.8D Nikon AF-S Nikkor 24-70mm f/2.8G ED +11 more
NilsDecker
NilsDecker Junior Member • Posts: 31
Re: Storage Solution for Large Photo Collection

Hi Wolff,

Thanks for sharing this, very helpful. I just finished reading the blog-post you wrote too which is even better (and coincidentally I literally got the same NAS in the mail today :-)).

How did you... figure out the NAS? It seems "a little" overwhelming at first. Which RAID configuration did you apply?

Thanks, Nils

BillyBobSenna
OP BillyBobSenna Senior Member • Posts: 2,049
Re: 700,000? How many giga-bytes?

Carey Brown wrote:

BillyBobSenna wrote:

Does anyone have a good storage recommendation for a large photo collection?

I have a higher end PC with a Lightroom catalog of about 700,000 photos spread across 5 internal hard drives. In addition, I have a Drobo 5C that serves as my backup for the 5 internal drives. I have no offsite backup.

I was thinking of moving my Lightroom catalog from the 5 internal drives to an external system and continuing to use the Drobo as a backup. I was hoping to have an external array device that appeared as 1 volume and was easier to expand as my collection grows.

Any recommendations?

700,000 photos is a heck of a lot of photos? Are you a professional?

Depends on your definition. I am paid for some events and at other times I shoot for clients in exchange for media access to events. Photography is not how I support myself.

You might want to do some ruthless culling to get that to a more manageable size.

Tell me about it. I do cull however I agree with your comment. Not enough time in the day.

How many giga-bytes of storage does it take up?

Approaching 20 TB

For me, I always want my entire collections of original photos and video on a single drive. Western Digital has drives all the way up to 10TB. I can't imagine your collection taking that much.

Larger internal drives may be the answer. WD actually has drives that are 12TB+.

A NAS can be 'a' backup but shouldn't be treated as 'the' backup (this assumes that all of your originals are on an internal drive). A power surge or fire could take out the entire unit. Additionally, if you have an 'oops' and accidentally delete a file, that deletion will probably be duplicated leaving you with no recoverable copy. It doesn't address the need for an off-site backup.

-- hide signature --

Bill - Beverly Hills, MI
Motorsports Photography
www.billgulkerphotography.com

 BillyBobSenna's gear list:BillyBobSenna's gear list
Nikon D500 Nikon D850 Nikon Z6 Nikon AF Nikkor 50mm f/1.8D Nikon AF-S Nikkor 24-70mm f/2.8G ED +11 more
Richard James Regular Member • Posts: 197
Re: Storage Solution for Large Photo Collection

BillyBobSenna wrote:

Richard James wrote:

Personally I would keep drive pairs (master and backup) all on separate devices, not shared devices.

Not sure what you mean here. Not use a Drobo? Why? If 1 drive in the Drobo goes bad, all data is still ok.

The "brand" does not matter. The point of a backup is at least 2 copies (preferably 3) on independent devices. 1 internal drive + 1 ext drive gives you 2 independent copies. Does not matter if it is a RAID based device or not.

RAID by itself is not 2 independent devices... if the whole system is inoperable.... you may be SOL.

My personal preference is to not put all my files on very large drives... if it goes, more is at risk, despite backups.

. I was hoping to have an external array device that appeared as 1 volume and was easier to expand as my collection grows.

If you do that you will have to go through your catalog and reset LR's drive pointers...folder by folder.

How long will it take you to go through your LR catalog folder by folder to reassign the drive letters from F, G, H, etc., to "X"?

Thanks for the feedback Richard. Do you have any drive recommendations?

Most of my HD's are WD, I have a few Segates. Also had Toshiba, Hitachi and IBM in the past. SSD's Samsung and Intel.

I've had a couple of failures (warranty replaced).

Richard

DerKeyser Regular Member • Posts: 439
Re: Storage Solution for Large Photo Collection

NilsDecker wrote:

Hi Wolff,

Thanks for sharing this, very helpful. I just finished reading the blog-post you wrote too which is even better (and coincidentally I literally got the same NAS in the mail today :-)).

How did you... figure out the NAS? It seems "a little" overwhelming at first. Which RAID configuration did you apply?

Thanks, Nils

Yes, right at start it looks a little daunting, but it’s not that bad once you get a few pointers. Hopefully these will help:

1: The NAS works with a concept called a storage pool. This is basically just the underlying disks RAID’ed together to provide gross space for different storage needs. Those needs are things like:

- Reserved space for the NAS OS itself
- Volumes (Filesystems) for your storage
- Snapshot space if you decide to snapshot volumes you created
So you need to create a storage pool from your disks in order to be able to create a volume (filesystem) for your data. When creating the storagepool you can select between different RAID levels. I only have two drives in my NAS, and hence i choose Raid 1 (Mirror). This allows one drive to fail without dataloss/problems.
If you have more drives, you could fx. look into Raid 6 which stripes data across 4 or more spindles of which two are used for “parity data” - IE: additional data calculated to allow for loss of up to two drives. So in Raid 6 you get your numbers of drives minus two’s capacity for userdata (fx: 6x8Tb equals 4x8Tb usable capacity).
You could also use Raid 5 which has only one drives capacity for parity data, but with todays HUGE drives, you do run a decent risk of not actually being able to rebuild a dead drive if you encounter a dead block on one of the remaining drives. If you make sure to have proper running backup, that could be an acceptable risk because of the drive capacity gained (or saved by buying one less drive).

2: Once your storage pool is created - lets in this example use 3x10Tb drives in Raid 5 - you will have 20Tb of net. Space (one drives capacity is used for parity in Raid5).
The NAS itself will reserve some 200Gb of the space for NAS features so you cannot use that. The remaining 19.7Tb can be used by you to create a volume on. There are two kinds of volumes:
- A thick volume that reserves all its space right away
- A Thin volume that reserves space from the pool when you add data, and give it back when you delete data.

IMPORTANT: The key point here is NOT to create a volume that uses up all the space in the storage pool because you want space for snapshots as well. So fx. Only create a volume of 17Tb to allow for snapshotspace.

It can be a little confusing that a Thin volume can be created much bigger that the actual free storagepool space. But it’s pretty neat as you can add drives to the pool one space starts to run out. So in this case we could make a 60Tb volume up front even though we only have 19.8Tb in the pool.

I will strongly recommend NOT overallocating like that since it requires you to monitor free space, and it can cause SERIOUS problems if something allows the pool to run out of space because data is added to the thin volume. So stick to creating a 17Tb volume - you can always increase its size once you have added more disks to the storage pool.

Once the 17Tb volume (Thick or Thin) is created, setup a snapshot schedule of that volume. fx. Create a snapshot every night at 02:00 and keep up to 14 snapshots (14 days).
Those snapshots will use the remaining space in the storagepool, and if the get bigger than the remaining 2.7Tb the oldest ones will delete automatically. For this very reason I can recommend using a Thin volume (but NOT over allocated) vs. A thick. That way snapshots can use the 2.7Tb remaining space + whatever is not used by the automatically growing/shrinking volume

Hopes all this makes sense

After your volume is ready and you have added data, make sure to download the APP called HBS 3 (Hybrid Backup Sync 3). This app allows you to setup automatic jobs that copy/backup or syncs data (I use Sync) to a cloud service on scheduled basis.
It also allows you to configure Local Backups via the One Touch Copy button on the front of the NAS (IE: When a USB drive is connected, pressing the button will sync all your data to the USB drive)

Good luck with your NAS - I’m sure you will love as much as I do once you get it all set up.

PS: There are a lot of APPs for the NAS that allows your images to be shared to clients around the house. Several uPnP/DNLA apps exist - fx. Photostation, but also more elaborate servers like PLEX are great. I use PLEX because that handles all my Music, Photos and Video in the same app on clients, and it allows offline sync of music/movies.

-- hide signature --

Happy Nikon Shooter
See my articles and galleries at: https://wolffmadsen.dk

Flashlight Veteran Member • Posts: 8,057
One caveat...
1

DerKeyser wrote:

1: It does Raid 6, so it can handle two drive failures in a set (fx 6x8Tb drives = 32Tb Usable space).

Many people with knowledge about RAID systems appear to advise against using more than 5 x 3TB drives in a RAID array. The rationale is that when a disk goes bad with larger arrays, during the repair cycle the stress on the other drives (which have the same age as the faulty one) is such that one or more are likely to fail as well.

I do have a few Synology NASes myself where one is at my daughters house in another city. We do both nightly automated backups to each other. In the long run this is much cheaper than a paid cloud service, especially with a large photo collection.

With Synology (do not know about QNap) the source of the backup must be on the internal drives, but the remote target can be a USB external drive attached to the remote Synology. So the Synology at my daughters house is a relatively cheap one-disk Synology with a large USB disk attached which holds a copy of my photo collection.

Another advantage of this strategy is that, if disaster strikes and I need the backup, I just drive to my daughters house and have all the images on a USB drive that I can plug directly into my computer. No need to slowly copy everything through the Internet.

-- hide signature --

Philip

DerKeyser Regular Member • Posts: 439
Re: One caveat...

Flashlight wrote:

DerKeyser wrote:

1: It does Raid 6, so it can handle two drive failures in a set (fx 6x8Tb drives = 32Tb Usable space).

Many people with knowledge about RAID systems appear to advise against using more than 5 x 3TB drives in a RAID array. The rationale is that when a disk goes bad with larger arrays, during the repair cycle the stress on the other drives (which have the same age as the faulty one) is such that one or more are likely to fail as well.

I do have a few Synology NASes myself where one is at my daughters house in another city. We do both nightly automated backups to each other. In the long run this is much cheaper than a paid cloud service, especially with a large photo collection.

With Synology (do not know about QNap) the source of the backup must be on the internal drives, but the remote target can be a USB external drive attached to the remote Synology. So the Synology at my daughters house is a relatively cheap one-disk Synology with a large USB disk attached which holds a copy of my photo collection.

Another advantage of this strategy is that, if disaster strikes and I need the backup, I just drive to my daughters house and have all the images on a USB drive that I can plug directly into my computer. No need to slowly copy everything through the Internet.

Yes, it is correct that in enterprise workloads it is not advised going above 5x3Tb in Raid5 due to the risk I mentioned. Also, that is Raid 5 - not Raid 6, and yes I know Raid 6 has it’s statistical challenges as well....

But note that the quality of cheap NAS/Always on disks have been increasing for years now, and the drives will in no way be exposed to a enterprise workload 24/7 on a private homeowner NAS. Most of the time they will likely be in “spindown” mode, and not really stressed at any point in their life.

So if you purchase quality NAS ready drives like WD Red or Seagate Ironwolf, I wouldn’t worry much about a Raid 6 failing. Raid 5 yes, but not 6. But it is REALLY important to buy the right kind of quality drives that support internal error correction and time limited error reporting. Standard desktop drives dies like flies when setup in a Raid array on a NAS. Also, I would recommend going with 5400Rpm disks like WD Red because of the way lower temperature, power usage and vibration levels. For sequential access to images/videos on a Raid array of disks it makes no performance difference anyways in a NAS limited til 1Gbe networking (112MB/s).

Whether having another NAS to replicate to or using a Cloud service is best, is a matter of personal choice. In my opinion A cloud copy combined with the 3rd copy being made by myself to a USB disk is FAR AND ABOVE more secure and better that having another NAS and Internet connection to setup/maintain. And I don’t believe another NAS is cheaper at any rate compared to cloud when you factor in renewal of hardware, power and so on.

The “File retention” mechanism of deleted or changed files a cloud service delivers alone (not mentioning the uptime and quality of their service compared to a 200$ NAS in a closet) makes it an instant winner for me.

But it obviously plays a big role that I want to follow good practice and have 3 copies of data, so the gains you think your USB drives at other location has, I have at home even faster with my own USB backup copy

-- hide signature --

Happy Nikon Shooter
See my articles and galleries at: https://wolffmadsen.dk

Flashlight Veteran Member • Posts: 8,057
Re: One caveat...

Wow, really stepped on your toes didn't I? Was not my intention

-- hide signature --

Philip

DerKeyser Regular Member • Posts: 439
Re: One caveat...

Flashlight wrote:

Wow, really stepped on your toes didn't I? Was not my intention

Oh no, not at all. Sorry if it came of a little “rant” like. Wasn’t my intention. I just wanted to argue that homeowners have a little more “leeway” with their solutions vs. Enterprise, and that I think it’s all about making the right decisions up front. That way you’ll have something resembeling the perfect setup in terms of data security and disaster tolerance - for not that big an investment. Trying to save those last 200-300$ on your storage really does come with some great caveats, risktaking and manual work

-- hide signature --

Happy Nikon Shooter
See my articles and galleries at: https://wolffmadsen.dk

NilsDecker
NilsDecker Junior Member • Posts: 31
Re: Storage Solution for Large Photo Collection

DerKeyser wrote:

NilsDecker wrote:

Hi Wolff,

Thanks for sharing this, very helpful. I just finished reading the blog-post you wrote too which is even better (and coincidentally I literally got the same NAS in the mail today :-)).

How did you... figure out the NAS? It seems "a little" overwhelming at first. Which RAID configuration did you apply?

Thanks, Nils

Yes, right at start it looks a little daunting, but it’s not that bad once you get a few pointers. Hopefully these will help:

Wow, amazing! It did/ does help a LOT!

1: The NAS works with a concept called a storage pool. This is basically just the underlying disks RAID’ed together to provide gross space for different storage needs. Those needs are things like:

- Reserved space for the NAS OS itself
- Volumes (Filesystems) for your storage
- Snapshot space if you decide to snapshot volumes you created
So you need to create a storage pool from your disks in order to be able to create a volume (filesystem) for your data. When creating the storagepool you can select between different RAID levels. I only have two drives in my NAS, and hence i choose Raid 1 (Mirror). This allows one drive to fail without dataloss/problems.
If you have more drives, you could fx. look into Raid 6 which stripes data across 4 or more spindles of which two are used for “parity data” - IE: additional data calculated to allow for loss of up to two drives. So in Raid 6 you get your numbers of drives minus two’s capacity for userdata (fx: 6x8Tb equals 4x8Tb usable capacity).
You could also use Raid 5 which has only one drives capacity for parity data, but with todays HUGE drives, you do run a decent risk of not actually being able to rebuild a dead drive if you encounter a dead block on one of the remaining drives. If you make sure to have proper running backup, that could be an acceptable risk because of the drive capacity gained (or saved by buying one less drive).

I have 4x4tb. Would you recommend RAID 5 or 6 in that case? 4TB IronWolf 5900 rpm SATA III 3.5" Internal NAS to be precise which seems to be what you recommend in your next comment?

2: Once your storage pool is created - lets in this example use 3x10Tb drives in Raid 5 - you will have 20Tb of net. Space (one drives capacity is used for parity in Raid5).
The NAS itself will reserve some 200Gb of the space for NAS features so you cannot use that. The remaining 19.7Tb can be used by you to create a volume on. There are two kinds of volumes:
- A thick volume that reserves all its space right away
- A Thin volume that reserves space from the pool when you add data, and give it back when you delete data.

IMPORTANT: The key point here is NOT to create a volume that uses up all the space in the storage pool because you want space for snapshots as well. So fx. Only create a volume of 17Tb to allow for snapshotspace.

I'm still in "play mode" so the way I had it set up was: 4x4tb in RAID 5, giving me 10.9tb usable storage and I created a volume of 10tb on it. I reserved some space for system applications etc.

Based on your recommendation, I now shrunk the actual volume to 7.9tb and allocated "guaranteed snapshot space" of 5%. Is that too low? I found somewhere in the documentation that the default is 20%.

It can be a little confusing that a Thin volume can be created much bigger that the actual free storagepool space. But it’s pretty neat as you can add drives to the pool one space starts to run out. So in this case we could make a 60Tb volume up front even though we only have 19.8Tb in the pool.

I will strongly recommend NOT overallocating like that since it requires you to monitor free space, and it can cause SERIOUS problems if something allows the pool to run out of space because data is added to the thin volume. So stick to creating a 17Tb volume - you can always increase its size once you have added more disks to the storage pool.

Once the 17Tb volume (Thick or Thin) is created, setup a snapshot schedule of that volume. fx. Create a snapshot every night at 02:00 and keep up to 14 snapshots (14 days).
Those snapshots will use the remaining space in the storagepool, and if the get bigger than the remaining 2.7Tb the oldest ones will delete automatically. For this very reason I can recommend using a Thin volume (but NOT over allocated) vs. A thick. That way snapshots can use the 2.7Tb remaining space + whatever is not used by the automatically growing/shrinking volume

So I think I understand. BUT. I currently set up the volume as a thick type, but, with a guaranteed snapshot space (see above, might increase to more than 5% which is what it's currently at). Isn't that the same then, somehow?

Hopes all this makes sense

After your volume is ready and you have added data, make sure to download the APP called HBS 3 (Hybrid Backup Sync 3). This app allows you to setup automatic jobs that copy/backup or syncs data (I use Sync) to a cloud service on scheduled basis.
It also allows you to configure Local Backups via the One Touch Copy button on the front of the NAS (IE: When a USB drive is connected, pressing the button will sync all your data to the USB drive)

I will look into all of that once I feel I've found the right setup of RAID/ thick vs. thin/ etc.

Thanks a MILLION, I really really appreciate it! 100x better than watching YT tutorials and reading manufacturer documentation.

Good luck with your NAS - I’m sure you will love as much as I do once you get it all set up.

PS: There are a lot of APPs for the NAS that allows your images to be shared to clients around the house. Several uPnP/DNLA apps exist - fx. Photostation, but also more elaborate servers like PLEX are great. I use PLEX because that handles all my Music, Photos and Video in the same app on clients, and it allows offline sync of music/movies.

Carey Brown
Carey Brown Senior Member • Posts: 1,940
700,000 = 20TB

BillyBobSenna wrote:

Carey Brown wrote:

BillyBobSenna wrote:

Does anyone have a good storage recommendation for a large photo collection?

I have a higher end PC with a Lightroom catalog of about 700,000 photos spread across 5 internal hard drives. In addition, I have a Drobo 5C that serves as my backup for the 5 internal drives. I have no offsite backup.

I was thinking of moving my Lightroom catalog from the 5 internal drives to an external system and continuing to use the Drobo as a backup. I was hoping to have an external array device that appeared as 1 volume and was easier to expand as my collection grows.

Any recommendations?

700,000 photos is a heck of a lot of photos? Are you a professional?

Depends on your definition. I am paid for some events and at other times I shoot for clients in exchange for media access to events. Photography is not how I support myself.

You might want to do some ruthless culling to get that to a more manageable size.

Tell me about it. I do cull however I agree with your comment. Not enough time in the day.

How many giga-bytes of storage does it take up?

Approaching 20 TB

Ok, I'm impressed. (I'll never whine about my increasing storage requirements again.) So, if you don't have time to cull them when you import them then when do you have time to keyword, search, view, or edit them?

Do you keep both a raw and jpg or tiff version? Any duplication? Do all files need to be backed up with the same priority?

For me, I always want my entire collections of original photos and video on a single drive. Western Digital has drives all the way up to 10TB. I can't imagine your collection taking that much.

Larger internal drives may be the answer. WD actually has drives that are 12TB+.

A NAS can be 'a' backup but shouldn't be treated as 'the' backup (this assumes that all of your originals are on an internal drive). A power surge or fire could take out the entire unit. Additionally, if you have an 'oops' and accidentally delete a file, that deletion will probably be duplicated leaving you with no recoverable copy. It doesn't address the need for an off-site backup.

As a wild-ass suggestion:

You could buy two NAS boxes with a lot of storage and find a (very) good friend or relative to house the other one. I think some of the brands support cross backups using the internet, presuming that both you and they have high speed internet and can tolerate hours of updates in the wee hours of the morning. In return, they could backup to their box and let it be copied to yours. Might require a commercial grade internet connection. On the other hand, how the hell would you tell that it''s working correctly?

At this point any solution will involve $e+4. What is it worth to you?

Sounds like 2x 12TB disks would be a way to start the process. Internal drives are relatively quick to access compared to NAS or other options.

-- hide signature --
mballe New Member • Posts: 20
Re: Storage Solution for Large Photo Collection

BillyBobSenna wrote:

Does anyone have a good storage recommendation for a large photo collection?

I have a higher end PC with a Lightroom catalog of about 700,000 photos spread across 5 internal hard drives. In addition, I have a Drobo 5C that serves as my backup for the 5 internal drives. I have no offsite backup.

I was thinking of moving my Lightroom catalog from the 5 internal drives to an external system and continuing to use the Drobo as a backup. I was hoping to have an external array device that appeared as 1 volume and was easier to expand as my collection grows.

Any recommendations?

I would take a look at Mylio. It can easily handle that amount of images, and give you access to all your photos on multiple devices.

https://focus.mylio.com/case-studies/mylio-helps-natural-exposures-manage-million-images

Keyboard shortcuts:
FForum MMy threads