Vignetting and flat frames ?

[stargazer ch]

While I am sincerely enjoying astrophotography with my iOptron SkyTracker and my Fuji X-T1 camera and I certainly have managed to get some reasonable results, I am yet sometimes struggling with vignetting and flat frames. So sometimes everything works perfectly and the results are just stunning but some other times I have circular patterns in my images.

My workflow is very simple (I like simple)

1) I use PixInsight @ Linux

2) I have a library of flat images for every focal length and aperture that are used to calibrate the lights - no darks, no bias - I took them on a day when it was so homogeneously foggy that I did not even need to bother to get a homogeneous artificial light source.

3) I then use StarAlignment, then ImageIntegration

4) In order to get rid of the Light Pollution and any gradients, I use either Automatic or Dynamic Background Extraction (ABE/DBE).

Using the above workflow I arrive at the below two images. You can immediately spot the lightly darker inside and the brighter outside (outside of a circle with an radius slightly smaller than the height of the image).

I have also used tools like SuperBias etc... and they help to some extent but they are introducing artefacts...

Integrated image with flat image applied (264x30s, XF50-140 mm f/2.8 @ 140 mm, f/4.0, 1600 ISO)

Same as above, but now with PixInsight's DBE (Dynamic Background Extraction) applied

When using lensfun and calculation lens profiles, I can see that the vignetting is quite heavy in the outer parts of the lens - see below

lens profile from 10 flat frames (XF50-140 mm f/2.8 @ 140 mm, f/4.0)

What could be producing the ring-like structure above ?

Can such a pattern caused by the lack of either bias or dark frames ?

How are other people in this forum dealing with vignetting ?

 

[rnclark]

What could be producing the ring-like structure above ?

A bad flat field is the most likely cause.

Can such a pattern caused by the lack of either bias or dark frames ?

Not bias. Bias is a single value for the entire frame. In theory it could be a weird pattern in the dark frames, but is unlikely. Does a dark frame show it before any applicatio of flat frames?

How are other people in this forum dealing with vignetting ?

I use a lens profile in photoshop.

Roger

 

[mermaidkiller]

I use a lens profile in photoshop.

Indeed, that is the best way. Unless you are using a 'lens' without a lens profile, such as a telescope. Then you should use flats, but they should be good. Bad flats mess up the image more than no flats at all.

Try to subtract the flat in Photoshop. Paste it as an extra layer above the image in PSD and stretch it so far that the center of the flat is 100% white. Then set the layer to 'Divide' instead of 'Normal' and play a bit with the Opacity slider and the levels slider of the flat frame layer itself, until you are satisfied.

Manual vignette correction (Filter => Lens Correction => Vignette) might work but not as good as flats or a lens profile.

The freeware Darktable app (an open source Lightroom variant) has more options on vignette correction. You can change the center, the amount of darkening, radius, etc.

 

[rnclark]

I use a lens profile in photoshop.

Indeed, that is the best way. Unless you are using a 'lens' without a lens profile, such as a telescope. Then you should use flats, but they should be good. Bad flats mess up the image more than no flats at all.

Try to subtract the flat in Photoshop. Paste it as an extra layer above the image in PSD and stretch it so far that the center of the flat is 100% white. Then set the layer to 'Divide' instead of 'Normal' and play a bit with the Opacity slider and the levels slider of the flat frame layer itself, until you are satisfied.

Manual vignette correction (Filter => Lens Correction => Vignette) might work but not as good as flats or a lens profile.

The freeware Darktable app (an open source Lightroom variant) has more options on vignette correction. You can change the center, the amount of darkening, radius, etc.

 

[stargazer ch]

I use a lens profile in photoshop.

Indeed, that is the best way. Unless you are using a 'lens' without a lens profile, such as a telescope. Then you should use flats, but they should be good. Bad flats mess up the image more than no flats at all.

Try to subtract the flat in Photoshop. Paste it as an extra layer above the image in PSD and stretch it so far that the center of the flat is 100% white. Then set the layer to 'Divide' instead of 'Normal' and play a bit with the Opacity slider and the levels slider of the flat frame layer itself, until you are satisfied.

Manual vignette correction (Filter => Lens Correction => Vignette) might work but not as good as flats or a lens profile.

The freeware Darktable app (an open source Lightroom variant) has more options on vignette correction. You can change the center, the amount of darkening, radius, etc.

--
Ricoh KR-5 ... Pentax ME Super ... Canon T90 ... ... ... 40d ... 7d ... 6d

Flats need to be removed by a multiply/divide, not subtraction. It must also be done on the linear data, not tone-mapped data.

If you are using photoshop and a telescope, make a lens profile. Adobe has a lens profile creator. The raw converter applies the flat field on the linear data before tone curve application.

Rawtherapee reportedly can use a flat field in the raw conversion, but I have not tried it.

Roger

Hi Roger,

thanks for your insights - much appreciated. I have now spent some time trying to figure this out. I initially thought about purchasing Adobe PS/LR and switching the whole work flow to Windows. But then I also explored RawTherapee and learnt how to apply the vignetting correction there but even that was still not good enough even if I was using the RAW internal parameters. So I went back and took another set of flat frames which eventually worked ! I already have a filter holder for my Bahtinov mask (Lonely Speck) so I simply wrapped two sheets of white paper around the plastic mask and put it in the filter holder to diffuse the light and avoid too much of a gradient. Using that the picture came out a lot better.

I am also working on a galaxy cluster image where the flat fields are so good that I can see the noise from the (missing) bias/dark frames.

New result but now with a "dark streak"



But as you can see - now I have secondary issue: In the latest flat frames I can see a light streak coming in from the top left corner of the image all the way to the center. You also can see this in the image above. This streak was not there in my earlier attempts. It only seems to affect daylight images. By changing the lens and shielding the camera case from daylight I can exclude this to be caused by the lens or some external light leak in the camera case. If I take a say 1 sec image in the dark, the whole background is totally smooth so the issue must somehow be triggered by the daylight.

I also could not see any visual damage when inspecting the sensor.



[ATTACH alt=""Flat Frame" based on 30 exposures (f=140 mm, f/4.0, ISO 800) ABE in PI applied"]1691575[/ATTACH]
"Flat Frame" based on 30 exposures (f=140 mm, f/4.0, ISO 800) ABE in PI applied



[ATTACH alt="Single frame of the above stack - ABE in PI applied - the "streak" is visible - although not as clear as above. "]1691576[/ATTACH]
Single frame of the above stack - ABE in PI applied - the "streak" is visible - although not as clear as above.

I am puzzled by the fact that this seems to be so along the diagonal and stopping pretty much in the center...

...or can you explain what possibly could cause such a thing ?

Many thanks in advance ?

 

[rnclark]

Hi Roger,

thanks for your insights - much appreciated. I have now spent some time trying to figure this out. I initially thought about purchasing Adobe PS/LR and switching the whole work flow to Windows. But then I also explored RawTherapee and learnt how to apply the vignetting correction there but even that was still not good enough even if I was using the RAW internal parameters. So I went back and took another set of flat frames which eventually worked ! I already have a filter holder for my Bahtinov mask (Lonely Speck) so I simply wrapped two sheets of white paper around the plastic mask and put it in the filter holder to diffuse the light and avoid too much of a gradient. Using that the picture came out a lot better.

I am also working on a galaxy cluster image where the flat fields are so good that I can see the noise from the (missing) bias/dark frames.

But as you can see - now I have secondary issue: In the latest flat frames I can see a light streak coming in from the top left corner of the image all the way to the center. You also can see this in the image above. This streak was not there in my earlier attempts. It only seems to affect daylight images. By changing the lens and shielding the camera case from daylight I can exclude this to be caused by the lens or some external light leak in the camera case. If I take a say 1 sec image in the dark, the whole background is totally smooth so the issue must somehow be triggered by the daylight.

I also could not see any visual damage when inspecting the sensor.

[ATTACH alt=""Flat Frame" based on 30 exposures (f=140 mm, f/4.0, ISO 800) ABE in PI applied"]1691575[/ATTACH]
"Flat Frame" based on 30 exposures (f=140 mm, f/4.0, ISO 800) ABE in PI applied

[ATTACH alt="Single frame of the above stack - ABE in PI applied - the "streak" is visible - although not as clear as above."]1691576[/ATTACH]
Single frame of the above stack - ABE in PI applied - the "streak" is visible - although not as clear as above.

I am puzzled by the fact that this seems to be so along the diagonal and stopping pretty much in the center...

...or can you explain what possibly could cause such a thing ?

Many thanks in advance ?

The diagonal line is very strange. It might be due to some reflection in your flat field setup. I am not sure what ABE in PI does, but the flat field image does not look right. The light fall-off should be uniform from center to edge, and not brighten in the corners. Figure 8 here: http://www.clarkvision.com/articles/technology-advancements-for-low-light-long-exposure-imaging/ shows a flat field. Or look at lenstip.com or other review site and look ad the vignetting data.

Roger

 

[tradesmith45]

I use a lens profile in photoshop.

Indeed, that is the best way. Unless you are using a 'lens' without a lens profile, such as a telescope. Then you should use flats, but they should be good. Bad flats mess up the image more than no flats at all.

Try to subtract the flat in Photoshop. Paste it as an extra layer above the image in PSD and stretch it so far that the center of the flat is 100% white. Then set the layer to 'Divide' instead of 'Normal' and play a bit with the Opacity slider and the levels slider of the flat frame layer itself, until you are satisfied.

Manual vignette correction (Filter => Lens Correction => Vignette) might work but not as good as flats or a lens profile.

The freeware Darktable app (an open source Lightroom variant) has more options on vignette correction. You can change the center, the amount of darkening, radius, etc.

--
Ricoh KR-5 ... Pentax ME Super ... Canon T90 ... ... ... 40d ... 7d ... 6d

Flats need to be removed by a multiply/divide, not subtraction. It must also be done on the linear data, not tone-mapped data.

If you are using photoshop and a telescope, make a lens profile. Adobe has a lens profile creator. The raw converter applies the flat field on the linear data before tone curve application.

Rawtherapee reportedly can use a flat field in the raw conversion, but I have not tried it.

Roger

Hi Roger,

thanks for your insights - much appreciated. I have now spent some time trying to figure this out. I initially thought about purchasing Adobe PS/LR and switching the whole work flow to Windows. But then I also explored RawTherapee and learnt how to apply the vignetting correction there but even that was still not good enough even if I was using the RAW internal parameters. So I went back and took another set of flat frames which eventually worked ! I already have a filter holder for my Bahtinov mask (Lonely Speck) so I simply wrapped two sheets of white paper around the plastic mask and put it in the filter holder to diffuse the light and avoid too much of a gradient. Using that the picture came out a lot better.

I am also working on a galaxy cluster image where the flat fields are so good that I can see the noise from the (missing) bias/dark frames.

New result but now with a "dark streak"

But as you can see - now I have secondary issue: In the latest flat frames I can see a light streak coming in from the top left corner of the image all the way to the center. You also can see this in the image above. This streak was not there in my earlier attempts. It only seems to affect daylight images. By changing the lens and shielding the camera case from daylight I can exclude this to be caused by the lens or some external light leak in the camera case. If I take a say 1 sec image in the dark, the whole background is totally smooth so the issue must somehow be triggered by the daylight.

I also could not see any visual damage when inspecting the sensor.

[ATTACH alt=""Flat Frame" based on 30 exposures (f=140 mm, f/4.0, ISO 800) ABE in PI applied"]1691575[/ATTACH]
"Flat Frame" based on 30 exposures (f=140 mm, f/4.0, ISO 800) ABE in PI applied

[ATTACH alt="Single frame of the above stack - ABE in PI applied - the "streak" is visible - although not as clear as above."]1691576[/ATTACH]
Single frame of the above stack - ABE in PI applied - the "streak" is visible - although not as clear as above.

I am puzzled by the fact that this seems to be so along the diagonal and stopping pretty much in the center...

...or can you explain what possibly could cause such a thing ?

Many thanks in advance ?

I don't use flats but do use Fuji & have a couple things for you to consider for figuring this out.

I remember reading somewhere that it's critical that flats be correctly exposed & be a bit darker exposure to work well. Take a look at the histogram of your flats.

Fuji has lens profile data built into the sidecar. Make sure the SW you are using for RAW development isn't automatically applying that or if it can be turned off. My experience w/ auto vignetting & distortion corrections is they tend to under correct & leave some residual. The combo of automatic correction & then application of a flat also auto corrected could produce the unique donut ring your getting especially after a strong stretch.

Lentip says Fuji auto vignetting corrections are only applied to jpg files so maybe this isn't an issue for you. LR seems to use this automatically.

The diagonal has to be coming from something outside the camera & lens. If bright lighting was used, its possible there are internal reflections inside your filter holder. Try flats using 2 layers of white tee shirt & uniform light source w/ no filter or holder. But I guessing you've already retaken the flats & have learned more.

Good luck!

 

[stargazer ch]

Yes, you're right - it's not a "normal" flat frame as such - ABE (Automatic Background Extractor is trying to remove any gradient it can find - the unusual pattern you can see in these "flat" frames is the residual left over from ABE. I just use it to bring out the actual problem better.

 

[stargazer ch]

Thanks for your pointers ! I have been studying various pieces of software in the last couple of weeks (PS, LR, Darktable, Rawtherapee, some Fuji tools) but neither of them is able to remove vignetting sufficiently accurate using the RAW file internal vignetting data.

My main tool currently really is PixInsight @ Linux (and sometimes i use rnclark's rnc-color-stretch) and I hope it will stay like that.

I don't think the diagonal line comes from outside of the camera. I now checked with 3 lenses and all show exactly the same pattern.

To me it looks like there is a reflection produced within the camera case.

I have been talking to my local Fuji dealer and he was as puzzled as me but he promised to forward the pictures with description to the Fuji Engineers to ask them for comment on what could be causing this.

Will keep you posted.