Computational Photography and Fujifilm

[kamerakiri]

Sorry to start a new thread that is in essence a continuation of the previous one. I wanted to write a few key points from Aaron Hockley's book, and figured the long response would be better suited to a new thread.

This book was suggested in the thread https://www.dpreview.com/forums/post/65733002

"The Computer Ate My Photos: Artificial Intelligence and the Future of Photography". It is a relatively short, easy to read book, available free on Kindle Unlimited.

I flicked through it in about an hour, and here are the key takeaways(with my comments in italics):

• AI in MILCs: Japanese vendors have implemented algorithm and AI oriented changes primarily for AF(face, eye, objects etc) and smart metering modes, as well as other features such as HDR, Panorama stitching etc that previously required external software. The rational expectation should be - this basic trend will continue. Cameras might identify more faces/objects simultaneously and jump between them quickly(R3 and Z9 already do a bit of this). The post processing tasks within a camera will increase as processing power increases. Don't expect any enormous leap here.
• Opening the eyes and AI driven shutter: I did not know that Apple Facetime manipulates the eyes and replaces them with AI modification in real time. Creepy. Facebook also has software they showcased some years back wherein they can "open" closed eyes using existing record of eyes from other Facebook photos. What about AI that can replace closed eyes during a blink, with AI eyes from other photos during the shoot, asks the author? Or replace "bad" eyes with "nice, moody" eyes from another photo? I can see potential use of software that triggers shutter to capture only when eyes are open(or upon seeing a smile etc), or sorts open and closed eyes automatically in folder. However AI eyes??? Do we expect camera to make AI abs for fitness shoots? Or AI muscles for playing Captain America or whatever new superhero they are running in cinemas?
• AI in color science: I won't write too long of an explanation. Suffice is to mention "Arsenal 2" developed by Ryan Stout. His "deep color" is an interesting concept. Since he is a photographer as well as a capable software engineer who has developed this tool himself - it actually cuts a lot of work without damaging the experience of photography. I don't know how well it works in practice though. Forum member forest dream will find this exciting, as I've seen him advocate for AI color profiles several times. Here are the links to explore:
  https://witharsenal.com/
• AI in metadata: This is for professional photographers who shoot high volume of clicks per event. I think it makes sense. Object recognition is pretty sophisticated these days, and it is not necessary to implement this in camera. Rather, it would be better implemented in accompanying editing app on desktop, with plenty of computing power at hand. AI based metadata classifies your photos using scene recognition and GPS data(if available), as well as based on event, the kind of sport, the kind of content(people/animals/architecture), on the basis of look, color and texture of the subject(day, night, sky, reflection, water, grass, woodland, fabric etc.). Author gives some examples, such as a wedding AI that tags the cake, kiss, flowers, people etc automatically. AI based tags can be done in camera itself. Of course most other cultures in world don't have wedding where cake, kiss or bouquet are present. I think third party apps will offer this earlier. Some degree of automatic tagging might be helpful for photo management. The special cases that vary from culture to culture may be addressed by third party apps.
• AI based flash: This is pretty easy and will become common quite soon. Basically a flash system that can analyze the distance to subject and walls, and automatically rotate to find the best angle. Relatively easy stuff and pretty useful as well.
• Advanced Auto mode: So that a professional camera can function as well as a smartphone and deliver pleasing images out of box. This implies automatic stacking and post processing to deliver best color according to subject. Most cameras already do this to some extent. It will only improve in future. A note on night sight/night mode in cameras: this AI driven mode from all major smartphone vendors(Apple/Google/Samsung) almost always produces false artifacts to bluff the impression of low noise and high sharpness. They replace noise with algorithm artifacts. Not sure if anyone would want this on MILCs - it veers into category of computer generated art. But then, some don't care if they get paid for it.
• AI based post-processing: Software chooses the "best" click, ranks "best" photos based upon AI trained on photographs that win competitions, selects "best" edits, can segment objects and create masks automatically etc. A lot of this is already here in third party apps. I don't see all this being offered on any camera though.
• Voice controlled camera: Self-explanatory. But, do you want to talk to your camera?(and send your data to unknown servers?) I can understand that this is a big deal in robotics.
• AI in restoration work: This is actually interesting. As you know, modern scanning techniques have been used to reveal many hidden layers of ancient historical items, such as old mummies and ancient paintings. AI in photographing historical items that can help guess how the painting might have looked, including generating original color and perhaps repairing minor scratches and wear marks. AI can also help reveal the fine details of brush and paintwork. It can automatically color B/W scans. This is a highly specialized task, but if GFX cameras bring some version of this in any future release, they can gain enormous popularity in art/archaeology conservation and restoration work.
• AI in industry: The vast majority of uses of AI are in the field of robotic devices and factory automation. A miniscule portion of this is relevant to photography in general, and MILCs in particular. Most AI related changes are presently being used to tag and sort data - this is because AI is still very computing power intensive and it takes a huge chunk of data to train it. Only American Internet giants and the government has such vast amount of data. Many people don't want Google to find its way into their camera(and fill it with ads).

The book covers many other use cases. You can read more on them. I've written the most important points above, and you can see, the list is not particularly huge.

The important thing to realize is that modern cameras are pretty capable already. Additional features may help the high volume professional cut the time spent before a computer(I'm sure this will happen), but there isn't much to add for a contemplative photographer. Pentax has done some interesting stuff for Astro shooters, and R3 EVF is convenient for sports. GFX can add some AI for art restoration and high quality scans. But these are again very special tasks. I haven't touched on video, but honestly, even video is limited today by only a few things - processor power, photography oriented camera batteries from a decade ago, lack of proper heat dissipation and contorted video/audio codec licensing. It is not easy to implement in hardware/GPU all variations of codecs that exist.

With that said - do you want any of these features in your Fujifilm camera? Add any ideas of your own as well(that makes sense for generic use case of MILCs).

 

[deednets]

Sorry to start a new thread that is in essence a continuation of the previous one. I wanted to write a few key points from Aaron Hockley's book, and figured the long response would be better suited to a new thread.

This book was suggested in the thread https://www.dpreview.com/forums/post/65733002

"The Computer Ate My Photos: Artificial Intelligence and the Future of Photography". It is a relatively short, easy to read book, available free on Kindle Unlimited.

I flicked through it in about an hour, and here are the key takeaways(with my comments in italics):

• AI in MILCs: Japanese vendors have implemented algorithm and AI oriented changes primarily for AF(face, eye, objects etc) and smart metering modes, as well as other features such as HDR, Panorama stitching etc that previously required external software. The rational expectation should be - this basic trend will continue. Cameras might identify more faces/objects simultaneously and jump between them quickly(R3 and Z9 already do a bit of this). The post processing tasks within a camera will increase as processing power increases. Don't expect any enormous leap here.
• Opening the eyes and AI driven shutter: I did not know that Apple Facetime manipulates the eyes and replaces them with AI modification in real time. Creepy. Facebook also has software they showcased some years back wherein they can "open" closed eyes using existing record of eyes from other Facebook photos. What about AI that can replace closed eyes during a blink, with AI eyes from other photos during the shoot, asks the author? Or replace "bad" eyes with "nice, moody" eyes from another photo? I can see potential use of software that triggers shutter to capture only when eyes are open(or upon seeing a smile etc), or sorts open and closed eyes automatically in folder. However AI eyes??? Do we expect camera to make AI abs for fitness shoots? Or AI muscles for playing Captain America or whatever new superhero they are running in cinemas?
• AI in color science: I won't write too long of an explanation. Suffice is to mention "Arsenal 2" developed by Ryan Stout. His "deep color" is an interesting concept. Since he is a photographer as well as a capable software engineer who has developed this tool himself - it actually cuts a lot of work without damaging the experience of photography. I don't know how well it works in practice though. Forum member forest dream will find this exciting, as I've seen him advocate for AI color profiles several times. Here are the links to explore:
  https://witharsenal.com/
• AI in metadata: This is for professional photographers who shoot high volume of clicks per event. I think it makes sense. Object recognition is pretty sophisticated these days, and it is not necessary to implement this in camera. Rather, it would be better implemented in accompanying editing app on desktop, with plenty of computing power at hand. AI based metadata classifies your photos using scene recognition and GPS data(if available), as well as based on event, the kind of sport, the kind of content(people/animals/architecture), on the basis of look, color and texture of the subject(day, night, sky, reflection, water, grass, woodland, fabric etc.). Author gives some examples, such as a wedding AI that tags the cake, kiss, flowers, people etc automatically. AI based tags can be done in camera itself. Of course most other cultures in world don't have wedding where cake, kiss or bouquet are present. I think third party apps will offer this earlier. Some degree of automatic tagging might be helpful for photo management. The special cases that vary from culture to culture may be addressed by third party apps.
• AI based flash: This is pretty easy and will become common quite soon. Basically a flash system that can analyze the distance to subject and walls, and automatically rotate to find the best angle. Relatively easy stuff and pretty useful as well.
• Advanced Auto mode: So that a professional camera can function as well as a smartphone and deliver pleasing images out of box. This implies automatic stacking and post processing to deliver best color according to subject. Most cameras already do this to some extent. It will only improve in future. A note on night sight/night mode in cameras: this AI driven mode from all major smartphone vendors(Apple/Google/Samsung) almost always produces false artifacts to bluff the impression of low noise and high sharpness. They replace noise with algorithm artifacts. Not sure if anyone would want this on MILCs - it veers into category of computer generated art. But then, some don't care if they get paid for it.
• AI based post-processing: Software chooses the "best" click, ranks "best" photos based upon AI trained on photographs that win competitions, selects "best" edits, can segment objects and create masks automatically etc. A lot of this is already here in third party apps. I don't see all this being offered on any camera though.
• Voice controlled camera: Self-explanatory. But, do you want to talk to your camera?(and send your data to unknown servers?) I can understand that this is a big deal in robotics.
• AI in restoration work: This is actually interesting. As you know, modern scanning techniques have been used to reveal many hidden layers of ancient historical items, such as old mummies and ancient paintings. AI in photographing historical items that can help guess how the painting might have looked, including generating original color and perhaps repairing minor scratches and wear marks. AI can also help reveal the fine details of brush and paintwork. It can automatically color B/W scans. This is a highly specialized task, but if GFX cameras bring some version of this in any future release, they can gain enormous popularity in art/archaeology conservation and restoration work.
• AI in industry: The vast majority of uses of AI are in the field of robotic devices and factory automation. A miniscule portion of this is relevant to photography in general, and MILCs in particular. Most AI related changes are presently being used to tag and sort data - this is because AI is still very computing power intensive and it takes a huge chunk of data to train it. Only American Internet giants and the government has such vast amount of data. Many people don't want Google to find its way into their camera(and fill it with ads).

The book covers many other use cases. You can read more on them. I've written the most important points above, and you can see, the list is not particularly huge.

The important thing to realize is that modern cameras are pretty capable already. Additional features may help the high volume professional cut the time spent before a computer(I'm sure this will happen), but there isn't much to add for a contemplative photographer. Pentax has done some interesting stuff for Astro shooters, and R3 EVF is convenient for sports. GFX can add some AI for art restoration and high quality scans. But these are again very special tasks. I haven't touched on video, but honestly, even video is limited today by only a few things - processor power, photography oriented camera batteries from a decade ago, lack of proper heat dissipation and contorted video/audio codec licensing. It is not easy to implement in hardware/GPU all variations of codecs that exist.

With that said - do you want any of these features in your Fujifilm camera? Add any ideas of your own as well(that makes sense for generic use case of MILCs).

A super reliable eye af plus a functioning level and hdr via curves, variants of raws based on your style.

A separate menu for style curves in conjunction with variable selective ND filters and maybe masking....

Basically anything adding to creative ways to use your camera. Without taking over. AF was helpful, eye AF and live tracking also. Super flat log files to create 20 stops of DR. Excluding the clumsy grading process..

Add to this an electronic image stabilization also taking into account rolling effects and 10 stops IS for 4K video (like MP3 vs CD, 8K over the top for most.

So, yeah, there's some food for thought, like "what have the Romans given us? Really???"

 

[Mar]

Just for me, and for my idea of computational Photography, would like to see :

• Sport,Animal,car autofocus
• High resolution shot (tripod and handheld)
• Live ND filter
• GND filters emulation in camera
• Live bulb photography.
• Astro photography tracking via GPS
• Focus stacking with final shot in camera

And software speaking, X Raw Studio for iOS and a better phone app with more features.

 

[kamerakiri]

A separate menu for style curves in conjunction with variable selective ND filters and maybe masking

Style curves will be cool.

Better stabilization is something that Fujifilm should deliver. I think they will improve it a lot in the X-Trans V generation.

 

[kamerakiri]

Just for me, and for my idea of computational Photography, would like to see :

• Sport,Animal,car autofocus
• High resolution shot (tripod and handheld)
• Live ND filter
• GND filters emulation in camera
• Live bulb photography.
• Astro photography tracking via GPS
• Focus stacking with final shot in camera

And software speaking, X Raw Studio for iOS and a better phone app with more features.

All vey good features that will help in acquiring photos of otherwise challenging subjects, without ruining the experience of photography. High Res Shot will be a big thing for X-Trans(it can be done, but the algorithms are likely to be slightly more complex).

I'm specially impressed that you included GND filter in the list. Not many think about this feature, but it can be very handy for achieving great landscape shots out of the box.

Fujifilm, and for that matter all camera companies, should put their processors on faster upgrade cycle. I hope with new generation of processors, we can see options such as focus stacking in camera (it is very resource intensive for anything above 20 shot stack for RAW file)

 

[Jerry-astro]

Just for purposes of making it easy for folks here to find and peruse content here, I think it makes more sense to try to collect more or less identical subject matter into a single thread. I'm not going to lock one of these threads, but for future posts such as this, I think it might be a lot more "user friendly" if more or less identical subject matter could be shared in a single thread... thus making it a bit easier for forum members to hold a "discussion" on that particular issue, rather than requiring interested folks to peruse multiple threads.

Again, this is not a forum rules issues, so please just take this as a recommendation from your friendly co-Mod.

 

[kamerakiri]

Just for purposes of making it easy for folks here to find and peruse content here, I think it makes more sense to try to collect more or less identical subject matter into a single thread. I'm not going to lock one of these threads, but for future posts such as this, I think it might be a lot more "user friendly" if more or less identical subject matter could be shared in a single thread... thus making it a bit easier for forum members to hold a "discussion" on that particular issue, rather than requiring interested folks to peruse multiple threads.

Again, this is not a forum rules issues, so please just take this as a recommendation from your friendly co-Mod.

Sure, thanks for the suggestion. I was conflicted about the same. I now realize that splitting the discussion was not a very efficient solution. I'll figure out a better context to start the thread in future.

 

[kamerakiri]

• Opening the eyes and AI driven shutter: I did not know that Apple Facetime manipulates the eyes and replaces them with AI modification in real time. Creepy.

All this AI stuff in smartphones is manipulating reality to give the appearance of better photos. MKBHD dropped a video on this. It summarizes this issue very well.

Here's the link:

This is not the direction I want computational photography to take in MILCs. Nobody knows what are the consequences of this AI driven warped reality.

 

[kamerakiri]

• What about AI that can replace closed eyes during a blink, with AI eyes from other photos during the shoot, asks the author? Or replace "bad" eyes with "nice, moody" eyes from another photo?

Crazy to think that Nokia Lumia phone from 2012 implemented a simple version of this - on much lower computing power available then compared to today.

 

[AstroVagabond]

Since you listed the most important points I no longer have a need to shell out some cash to purchase the authors book. Much appreciated.

Not sure the author is appreciative of you causing him to lose a sale but that is between you and them.

 

[DarnGoodPhotos]

Sorry to start a new thread that is in essence a continuation of the previous one. I wanted to write a few key points from Aaron Hockley's book, and figured the long response would be better suited to a new thread.

This book was suggested in the thread https://www.dpreview.com/forums/post/65733002

"The Computer Ate My Photos: Artificial Intelligence and the Future of Photography". It is a relatively short, easy to read book, available free on Kindle Unlimited.

I flicked through it in about an hour, and here are the key takeaways(with my comments in italics):

• AI in MILCs: Japanese vendors have implemented algorithm and AI oriented changes primarily for AF(face, eye, objects etc) and smart metering modes, as well as other features such as HDR, Panorama stitching etc that previously required external software. The rational expectation should be - this basic trend will continue. Cameras might identify more faces/objects simultaneously and jump between them quickly(R3 and Z9 already do a bit of this). The post processing tasks within a camera will increase as processing power increases. Don't expect any enormous leap here.
• Opening the eyes and AI driven shutter: I did not know that Apple Facetime manipulates the eyes and replaces them with AI modification in real time. Creepy. Facebook also has software they showcased some years back wherein they can "open" closed eyes using existing record of eyes from other Facebook photos. What about AI that can replace closed eyes during a blink, with AI eyes from other photos during the shoot, asks the author? Or replace "bad" eyes with "nice, moody" eyes from another photo? I can see potential use of software that triggers shutter to capture only when eyes are open(or upon seeing a smile etc), or sorts open and closed eyes automatically in folder. However AI eyes??? Do we expect camera to make AI abs for fitness shoots? Or AI muscles for playing Captain America or whatever new superhero they are running in cinemas?
• AI in color science: I won't write too long of an explanation. Suffice is to mention "Arsenal 2" developed by Ryan Stout. His "deep color" is an interesting concept. Since he is a photographer as well as a capable software engineer who has developed this tool himself - it actually cuts a lot of work without damaging the experience of photography. I don't know how well it works in practice though. Forum member forest dream will find this exciting, as I've seen him advocate for AI color profiles several times. Here are the links to explore:
  https://witharsenal.com/
• AI in metadata: This is for professional photographers who shoot high volume of clicks per event. I think it makes sense. Object recognition is pretty sophisticated these days, and it is not necessary to implement this in camera. Rather, it would be better implemented in accompanying editing app on desktop, with plenty of computing power at hand. AI based metadata classifies your photos using scene recognition and GPS data(if available), as well as based on event, the kind of sport, the kind of content(people/animals/architecture), on the basis of look, color and texture of the subject(day, night, sky, reflection, water, grass, woodland, fabric etc.). Author gives some examples, such as a wedding AI that tags the cake, kiss, flowers, people etc automatically. AI based tags can be done in camera itself. Of course most other cultures in world don't have wedding where cake, kiss or bouquet are present. I think third party apps will offer this earlier. Some degree of automatic tagging might be helpful for photo management. The special cases that vary from culture to culture may be addressed by third party apps.
• AI based flash: This is pretty easy and will become common quite soon. Basically a flash system that can analyze the distance to subject and walls, and automatically rotate to find the best angle. Relatively easy stuff and pretty useful as well.
• Advanced Auto mode: So that a professional camera can function as well as a smartphone and deliver pleasing images out of box. This implies automatic stacking and post processing to deliver best color according to subject. Most cameras already do this to some extent. It will only improve in future. A note on night sight/night mode in cameras: this AI driven mode from all major smartphone vendors(Apple/Google/Samsung) almost always produces false artifacts to bluff the impression of low noise and high sharpness. They replace noise with algorithm artifacts. Not sure if anyone would want this on MILCs - it veers into category of computer generated art. But then, some don't care if they get paid for it.
• AI based post-processing: Software chooses the "best" click, ranks "best" photos based upon AI trained on photographs that win competitions, selects "best" edits, can segment objects and create masks automatically etc. A lot of this is already here in third party apps. I don't see all this being offered on any camera though.
• Voice controlled camera: Self-explanatory. But, do you want to talk to your camera?(and send your data to unknown servers?) I can understand that this is a big deal in robotics.
• AI in restoration work: This is actually interesting. As you know, modern scanning techniques have been used to reveal many hidden layers of ancient historical items, such as old mummies and ancient paintings. AI in photographing historical items that can help guess how the painting might have looked, including generating original color and perhaps repairing minor scratches and wear marks. AI can also help reveal the fine details of brush and paintwork. It can automatically color B/W scans. This is a highly specialized task, but if GFX cameras bring some version of this in any future release, they can gain enormous popularity in art/archaeology conservation and restoration work.
• AI in industry: The vast majority of uses of AI are in the field of robotic devices and factory automation. A miniscule portion of this is relevant to photography in general, and MILCs in particular. Most AI related changes are presently being used to tag and sort data - this is because AI is still very computing power intensive and it takes a huge chunk of data to train it. Only American Internet giants and the government has such vast amount of data. Many people don't want Google to find its way into their camera(and fill it with ads).

The book covers many other use cases. You can read more on them. I've written the most important points above, and you can see, the list is not particularly huge.

The important thing to realize is that modern cameras are pretty capable already. Additional features may help the high volume professional cut the time spent before a computer(I'm sure this will happen), but there isn't much to add for a contemplative photographer. Pentax has done some interesting stuff for Astro shooters, and R3 EVF is convenient for sports. GFX can add some AI for art restoration and high quality scans. But these are again very special tasks. I haven't touched on video, but honestly, even video is limited today by only a few things - processor power, photography oriented camera batteries from a decade ago, lack of proper heat dissipation and contorted video/audio codec licensing. It is not easy to implement in hardware/GPU all variations of codecs that exist.

With that said - do you want any of these features in your Fujifilm camera? Add any ideas of your own as well(that makes sense for generic use case of MILCs).

A super reliable eye af plus a functioning level and hdr via curves, variants of raws based on your style.

A separate menu for style curves in conjunction with variable selective ND filters and maybe masking....

Basically anything adding to creative ways to use your camera. Without taking over. AF was helpful, eye AF and live tracking also. Super flat log files to create 20 stops of DR. Excluding the clumsy grading process..

Add to this an electronic image stabilization also taking into account rolling effects and 10 stops IS for 4K video (like MP3 vs CD, 8K over the top for most.

So, yeah, there's some food for thought, like "what have the Romans given us? Really???"

Now I have to conjugate "Robots go home".

 

[DarnGoodPhotos]

• What about AI that can replace closed eyes during a blink, with AI eyes from other photos during the shoot, asks the author? Or replace "bad" eyes with "nice, moody" eyes from another photo?

Crazy to think that Nokia Lumia phone from 2012 implemented a simple version of this - on much lower computing power available then compared to today.

I thought the Nokia he mentioned took a burst and let you scroll between the individual photos.

Eye replacement, like a lot of CP, automates what was a manual process. Some CP features go deeper than just simply stacking to help eliminate alignment to motion artifacts. Then there is CP where you let the AI decide certain things.

However creepy it might appear, things like Apple's Eye Contact do solve the problem during vide chat of cameras being located to the side of, or above, your line of sight.

 

[rla1022]

Just for me, and for my idea of computational Photography, would like to see :

• Sport,Animal,car autofocus
• High resolution shot (tripod and handheld)
• Live ND filter
• GND filters emulation in camera
• Live bulb photography.
• Astro photography tracking via GPS
• Focus stacking with final shot in camera

And software speaking, X Raw Studio for iOS and a better phone app with more features.