Reading mode: Light Dark

Desktop PC processing large numbers of images

Go to last post
Ephemeris

Ephemeris

Veteran Member
Messages
6,916
Solutions
2
Reaction score
1,292
Location
West Yorkshire, UK
Hi folks.

We often find ourselves processing a large number of images. The source is some canon Eos R5 cameras. Usually shoot JPEG + RAW and often just process the JPEGS.

Images are often at ISO 6400 or higher.

They are then processed in Topaz Denoise AI (will process jpeg and raw).

It is extremely time consuming to batch process 400 to 600 images per day.

Is there a way to process over a cluster?

If we were to build a new PC to speed this job up what should we be looking at building? (It must run Windows or a variant of Linux)

Many thanks.
 
Sean Nelson said:
Ephemeris said:
It is extremely time consuming to batch process 400 to 600 images per day.
Click to expand...
Are you concerned about wallclock time or man-hours? If the former, then some sort of upgrade is what you want. If the latter, then some sort of automation.
Click to expand...
Time it takes to process all images.
 
DQ03 said:
Ephemeris said:
Thank you.

So I think you went for the 3050? What drove you to that choice?
Click to expand...
Decent price ($339) for a 8 GB graphic card.
Click to expand...
Also, the components were picked based on equivalent specs of a Dell computer that my cousin really loves to have but cannot afford. His budget was $1,800 max.

PS. I chose 2x1 TB SSD instead of 1x2 TB, just in case of failure.

00d5d557cb4147aebc559cdb5d45934e.jpg

--
https://flickr.com/photos/10121023@N08/albums
 
Last edited:
DQ03 said:
DQ03 said:
Ephemeris said:
Thank you.

So I think you went for the 3050? What drove you to that choice?
Click to expand...
Decent price ($339) for a 8 GB graphic card.
Click to expand...
Also, the components were picked based on equivalent specs of a Dell computer that my cousin really loves to have but cannot afford. His budget was $1,800 max.

PS. I chose 2x1 TB SSD instead of 1x2 TB, just in case of failure.

00d5d557cb4147aebc559cdb5d45934e.jpg
Click to expand...
Also, you used the RTX 3050, but the Dell has the RTX 3070 so that helped you to keep the cost lower. I assume you used 32gb DDR5 RAM in your build too?

--
Henry Richardson
 
Henry Richardson said:
DQ03 said:
DQ03 said:
Ephemeris said:
Thank you.

So I think you went for the 3050? What drove you to that choice?
Click to expand...
Decent price ($339) for a 8 GB graphic card.
Click to expand...
Also, the components were picked based on equivalent specs of a Dell computer that my cousin really loves to have but cannot afford. His budget was $1,800 max.

PS. I chose 2x1 TB SSD instead of 1x2 TB, just in case of failure.

00d5d557cb4147aebc559cdb5d45934e.jpg
Click to expand...
Also, you used the RTX 3050, but the Dell has the RTX 3070 so that helped you to keep the cost lower. I assume you used 32gb DDR5 RAM in your build too?
Click to expand...
I do not know much about 3070 and how much better it is compared to 3050 although both has 8 GB, but am wondering if it’s worth $700 more? And yes, the RAM is DDR5, 5200 MHz, faster than the Dell.

--
https://flickr.com/photos/10121023@N08/albums
 
Last edited:
DQ03 said:
Henry Richardson said:
DQ03 said:
DQ03 said:
Ephemeris said:
Thank you.

So I think you went for the 3050? What drove you to that choice?
Click to expand...
Decent price ($339) for a 8 GB graphic card.
Click to expand...
Also, the components were picked based on equivalent specs of a Dell computer that my cousin really loves to have but cannot afford. His budget was $1,800 max.

PS. I chose 2x1 TB SSD instead of 1x2 TB, just in case of failure.

00d5d557cb4147aebc559cdb5d45934e.jpg
Click to expand...
Also, you used the RTX 3050, but the Dell has the RTX 3070 so that helped you to keep the cost lower. I assume you used 32gb DDR5 RAM in your build too?
Click to expand...
I do not know much about 3070 and how much better it is compared to 3050 but is it worth $700 more? And yes, the RAM is DDR5, 5200 MHz, faster than the Dell.
Click to expand...
Yeah, for what I use a computer for I would have gone for the 3050 also. But using the 3050 rather than the 3070 accounts for a lot of your cost savings.

--
Henry Richardson
 
Member said:
Henry Richardson wrote.
Click to expand...
Member said:
Yeah, for what I use a computer for I would have gone for the 3050 also. But using the 3050 rather than the 3070 accounts for a lot of your cost savings.
Click to expand...
Me too! I could have used 3070, that would have cost him $300 more but he’ d still have been $400 ahead of the Dell.

I’m waiting for the price of graphics cards to come down further to build a similar one for myself, just for photo editing, no gaming.
 
DQ03 said:
DQ03 said:
Ephemeris said:
Thank you.

So I think you went for the 3050? What drove you to that choice?
Click to expand...
Decent price ($339) for a 8 GB graphic card.
Click to expand...
Also, the components were picked based on equivalent specs of a Dell computer that my cousin really loves to have but cannot afford. His budget was $1,800 max.

PS. I chose 2x1 TB SSD instead of 1x2 TB, just in case of failure.

00d5d557cb4147aebc559cdb5d45934e.jpg
Click to expand...
Backup is the answer to risk of failure, not spreading data around. But you should have two SSDs anyway, one for all the temporary storage: the virtual RAM swap file, Lightroom cache and previews, etc. Better use of bus BW, and none of that needs to be backed up.
 
Chris Noble said:
Backup is the answer to risk of failure, not spreading data around. But you should have two SSDs anyway, one for all the temporary storage: the virtual RAM swap file, Lightroom cache and previews, etc. Better use of bus BW, and none of that needs to be backed up.
Click to expand...
I’ve read many of your useful posts. Thanks.

I plan to clone his c drive to the d drive as a backup drive in case the former fails. Although he has spare room on both drives, he will be using external drives for photo storage.

Hope I’m not off track too far, if you don’t mind, how do I set a drive for 1) virtual RAM swap and 2) LR cache and previews. Are they automatic or you have to set them manually. I have not kept myself abreast of soft/hardware setups. I’m currently using LR 6.14 and my pc was built about a decade ago (i7, 16 gb ram, 4 gb graphic card, c: 1 tb SATA ssd, d: 2 tb hhd.)
 
I'm not Chris Noble, and I don't use Lightroom.

However, Photoshop allows you to set a "scratch" drive. I assume that Lightroom is the same.

Windows permits you to choose a drive that's used for virtual memory. By default, it's the OS drive.
 
Last edited:
Different markets. A gamer dropping 2K on a GPU is often just bragging to his friends. A workstation card is making money for the buyer.
 
BobKnDP said:
I'm not Chris Noble, and I don't use Lightroom.

However, Photoshop allows you to set a "scratch" drive. I assume that Lightroom is the same.

Windows permits you to choose a drive that's used for virtual memory. By default, it's the OS drive.
Click to expand...
Thanks Bob for your input. I don’t have Photoshop but will look into LR to see the performance difference b/w drives for “scratch” and will leave the 1 TB OS drive as a default source for virtual memory.
 
Ephemeris said:
… If we were to build a new PC to speed this job up what should we be looking at building? (It must run Windows or a variant of Linux)
Click to expand...
I forgot to mention that I test drove the pc I built for my cousin and compared the time it took to AI sharpen a photo with my 10 years old+ pc. The result is staggering: 15 seconds vs. 20 minutes. He’s as happy as a clam!

--
https://flickr.com/photos/10121023@N08/albums
 
Last edited:
DQ03 said:
Ephemeris said:
… If we were to build a new PC to speed this job up what should we be looking at building? (It must run Windows or a variant of Linux)
Click to expand...
I forgot to mention that I test drove the pc I built for my cousin and compared the time it took to AI sharpen a photo with my 10 years old+ pc. The result is staggering: 15 seconds vs. 20 minutes. He’s as happy as a clam!
Click to expand...
 
Ephemeris said:
DQ03 said:
Ephemeris said:
… If we were to build a new PC to speed this job up what should we be looking at building? (It must run Windows or a variant of Linux)
Click to expand...
I forgot to mention that I test drove the pc I built for my cousin and compared the time it took to AI sharpen a photo with my 10 years old+ pc. The result is staggering: 15 seconds vs. 20 minutes. He’s as happy as a clam!
Click to expand...
Thanks for the data

I don't suppose you tried it with GFX acceleration turned off?

Just to see why / where it was 80x faster?
Click to expand...
Beats me! I wouldn’t know where to turn “GFX acceleration” on or off. The new pc was picked up 2 days ago, I wish I still have it to find/try out more.

I don’t have all the specific data but think the processing time is a lot shorter due to a combination of many factors including newer, faster, more powerful and more advanced CPU, GPU, DDR5 RAM, SSD, DDR5 Mo Bo, Intel chipset, … compared to my pc which is really old.

--
https://flickr.com/photos/10121023@N08/albums
 
Last edited:
Robert Zanatta said:
3070 is a lot faster. Whether it's worth the extra cost depends on a lot of factors.
Click to expand...
It must be but I just don’t know how much faster/better for at least $300 more. The one I installed is MSI, it has 8 GB GDDR6, same as the 3070 used in the Dell (I do not know what make). I start to gather that the same number of GB is not enough, why?

Anyway, the pc I built has faster RAM (5200 MHz) than the Dell (4400 MHz), hope the higher speed compensates somehow for a lesser GPU.

--
https://flickr.com/photos/10121023@N08/albums
 
Last edited:
Ephemeris said:
Hi folks.

We often find ourselves processing a large number of images. The source is some canon Eos R5 cameras. Usually shoot JPEG + RAW and often just process the JPEGS.

Images are often at ISO 6400 or higher.

They are then processed in Topaz Denoise AI (will process jpeg and raw).

It is extremely time consuming to batch process 400 to 600 images per day.

Is there a way to process over a cluster?
Click to expand...
Wow, you don't get people talking about clusters much these days (I was System Manager of a Vaxcluster back in the late 1980s).

Speaking broadly, clusters were loosely linked discrete processors in their own boxes sharing a storage array, but each with its own discrete memory and tasks - the benefit was extra resilience, with load sharing being done at login - you could always add a new node to the cluster without having to bring the whole thing down.

You could do similar running two (or more) PCs off a Network Storage device, but it wouldn't be automatic - you would have to choose 200-300 images to process on one and do the rest on the other. It would probably get close to halving the process time.

In practice modern multi-core processors with more dynamic balancing are far more practical.
Ephemeris said:
If we were to build a new PC to speed this job up what should we be looking at building? (It must run Windows or a variant of Linux)

Many thanks.
Click to expand...
Have you considered other software?

You can get a trial on stuff like DXO PL5 or Capture 1 22 (or variuous other good options). Some software exploit extra cores better than others, you might find that alternative software runs faster with similar results.

The effectiveness of software in using the cores available on various types of processor varies, hence clock speed tends to be king, but software that effectively uses the cores will fly in comparison to software that does not use them well.

--
Save a life, become a stem-cell donor.
Hello to Jason Isaacs!
https://bobjanes.smugmug.com/PoTB/
Please respect a BY-NC-ND cc licence on all my public internet images
 
Last edited:
You must log in or register to reply here.

Keyboard shortcuts

f
Forum
m
My threads
Mobile site
About
Editorial content
Cameras & Lenses
Community
All content, design, and layout are Copyright © 1998–2025 Digital Photography Review All Rights Reserved.
Reproduction in whole or part in any form or medium without specific written permission is prohibited.
When you use DPReview links to buy products, the site may earn a commission.
©GPS Media - Guides, Products, Services.
Back
Top