satellite and hot pixel track suppression in software

Started 2 months ago | Discussions
hha Contributing Member • Posts: 653
satellite and hot pixel track suppression in software

Airplane, satellite tracks and hot pixels are problem in long time exposure astrophotography. Most cameras have a dark frame subtraction option to eliminate hot pixels, but at the expense of loosing 50% of the exposure time. Most image processing software include an option to suppress artifacts in a long sequence of images. Some have the reputation of being “star eaters”. Here is how I solved this problem with an example using the “Accumulation 3.5” option in SEQUATOR.

I took a 2 hour exposure (120 shots at 60 sec each) using a f=135mm lens at f/4 of the Leo constellation. My Nikon Z6 was mounted on a SkyTracker to eliminate star trails. Rather than waiting for a satellite track, I intentionally misaligned the pole axis by 0.5 degrees. This produces 0.26 degree long star trail in 2 hours. The result of a simple star registration of the 120 images is shown on the left below. This is 1x1 degree cropped from the 10 x 15 degree field, zoomed in at 400% to make the pixels visible.

The stars are perfectly registered. The most prominent star on the lower left is +6.7 (HD97937) in Leo. The galaxy in the upper right is NGC3593 (+11.8). The faintest stars are +15. The bluish trace is produced by a hot pixel. It shows the worm period and a 3 pixel peak-to-peak worm error. Since each pixel of the camera with the 135 mm lens subtends 9”, the peak-to-peak worm error is 27 arcsec. This is a well known limitation of the SkyTracker, but it has almost no effect on the image quality with 60 second exposures. The hot pixel track is not straight and uniform, indicating that the mount flexed during the 2 hours.

On the right above is the same 120 images, but this time registered with the 3.5 sigma accumulation option in Sequator. Most of the effect of the hot pixel has disappeared, and there is no indication of the faintest star being effected. If the polar alignment had been near perfect, the hot pixel would have looked like a faint star in an unexpected location. The lesson from this (for me) is that a little polar misalignment can be helpful in eliminating hot pixels and avoiding wasted time to identify "new" supernovae.

There are various algorithms to accomplish this. The algorithm used by Sequator increases the data processing time from 5 minutes to 11 minutes for the 120 images. If anybody knows the details of the Sequator algorithm, please share.

There may be better and faster algorithms for this purpose. Please share your experience with 3 sigma editing with other star registration software.

Cheer,

hha

 hha's gear list:hha's gear list
Olympus TG-630 iHS Nikon D3200 Nikon D3300 Nikon D7200 Nikon AF Nikkor 24mm f/2.8D +1 more
Nikon Z6
If you believe there are incorrect tags, please send us this post using our feedback form.
Alen K
Alen K Senior Member • Posts: 1,298
Re: satellite and hot pixel track suppression in software

Sequator uses what is often called kappa-sigma clipping or rejection in its "select best pixels" accumulation mode, in three passes according to the manual. The rejection setting (related to standard deviation, I believe) is 2.0 by default. By calling it "accumulation 3.5" I assume you are changing the setting to 3.5. Different images can need different settings but so far 2.0 has always worked for me. (Actually, I think the scale only goes up to 3.3 from a minimum of 1.4.)

As long as I have at least eight light frames in my stack, I never use anything else. (My largest stack so far was nearly 400 frames.) If I do have less than eight frames, I also select "reduce dynamic noises." If I have less than four frames, which Sequator's manual says is the minimum required for that accumulation mode, I do not even try to activate "select best pixels." (I'm not sure the program will let you.)

Your worm track is one of the clearest I have yet seen. As you have discovered, those can be invaluable for evaluating sources of tracking error. Although intentionally offsetting the polar alignment to induce declination drift combined with kappa-sigma stacking is one way to solve those problems, I would encourage you to try to reduce them at the source.

And instead of offsetting the polar alignment, I recommend doing random dithering. You may find, as many others have experienced, that PA offset alone will cause walking noise in the background. Random dithering significantly reduces that as well as satisfying the conditions for kappa-sigma clipping. I always do random dithering but it does take more than eight frames; at least a dozen different, random positions and the more the better.

BTW, "select best pixels" is slower both because of the multiple iterations and because it writes temporary files to your computer's drive. If you have an SSD, be sure to specify a directory on that as the location for your output image rather than a hard-drive. SSD's are, of course, much faster for both reads and writes.

OP hha Contributing Member • Posts: 653
Re: satellite and hot pixel track suppression in software

Alen K wrote:

Sequator uses what is often called kappa-sigma clipping or rejection in its "select best pixels" accumulation mode, in three passes according to the manual. The rejection setting (related to standard deviation, I believe) is 2.5 by default. By calling it "accumulation 3.5" I assume you are changing the setting to 3.5. Different images can need different settings but so far 2.5 has always worked for me.

Alen:

Thank you for the detailed response.

The reliable determination of the standard deviation for each pixel is the trickiest part of the clipping algorithm. Pretty reliable with 100 frames, nearly impossible with 10 frames. I select the mildest filter (3.5) because I wanted to avoid clipping out faint star signals, which happens frequently with the 2.5 threshold.

As long as I have at least eight images in my stack, I never use anything else. (My largest stack so far was nearly 400 frames.) If I do have less than eight frames, I also select "reduce dynamic noises." If I have less than four frames, which Sequator's manual says is the minimum required for that accumulation mode, I do not activate "select best pixels."

With only eight frames the determination of sigma and what to clip out becomes very hazy. Faint stars disappear. I never use clipping with less than 60 frames.

Your worm track is one of the clearest I have yet seen. As you have discovered, those can be invaluable for evaluating sources of tracking error. Although intentionally offsetting the polar alignment to induce declination drift combined with kappa-sigma stacking is one way to solve those problems, I would encourage you to try to reduce them at the source.

I agree. When Polaris is visible, my alignment with the Polaris scope alone is much better than 0.5 degree. I have taken the Skytracker to camp sites high in the mountains, where Polaris may be hidden by trees or mountains and I can only align with a magnetic compass.

I  did this test to evaluate

1. the SkyTracker tracking error and discovered that the flexing error in the tripod is significant. The camera with the lens weighs only 1.5 kg. No counterweights were used.

2. how much pole axis alignment error I could get away with.

And instead of offsetting the polar alignment, I recommend doing random dithering. You may find, as many others have experienced, that PA offset alone will cause walking noise in the background.

The walking noise ( due to gaussian noise spikes) is also virtually eliminated by sigma clipping.

Random dithering significantly reduces that as well as satisfying the conditions for kappa-sigma clipping. I always do random dithering but it does take more than eight frames; at least a dozen different, random positions and the more the better.

Interesting, but random dithering is not an option with the SkyTracker.

BTW, "select best pixels" is slower both because of the multiple iterations and because it writes temporary files to your computer's drive. If you have an SSD, be sure to specify a directory on that as the location for your output image rather than a hard-drive. SSD's are, of course, much faster for both reads and writes.

I write the huge scratch files from each frame into the SSH on the motherboard.

Have you tried the worm error and flexing with your setup?

Cheers.

hha

 hha's gear list:hha's gear list
Olympus TG-630 iHS Nikon D3200 Nikon D3300 Nikon D7200 Nikon AF Nikkor 24mm f/2.8D +1 more
Jared Willson Senior Member • Posts: 1,203
Re: satellite and hot pixel track suppression in software

hha wrote:

Airplane, satellite tracks and hot pixels are problem in long time exposure astrophotography. Most cameras have a dark frame subtraction option to eliminate hot pixels, but at the expense of loosing 50% of the exposure time. Most image processing software include an option to suppress artifacts in a long sequence of images. Some have the reputation of being “star eaters”. Here is how I solved this problem with an example using the “Accumulation 3.5” option in SEQUATOR.

I took a 2 hour exposure (120 shots at 60 sec each) using a f=135mm lens at f/4 of the Leo constellation. My Nikon Z6 was mounted on a SkyTracker to eliminate star trails. Rather than waiting for a satellite track, I intentionally misaligned the pole axis by 0.5 degrees. This produces 0.26 degree long star trail in 2 hours. The result of a simple star registration of the 120 images is shown on the left below. This is 1x1 degree cropped from the 10 x 15 degree field, zoomed in at 400% to make the pixels visible.

The stars are perfectly registered. The most prominent star on the lower left is +6.7 (HD97937) in Leo. The galaxy in the upper right is NGC3593 (+11.8). The faintest stars are +15. The bluish trace is produced by a hot pixel. It shows the worm period and a 3 pixel peak-to-peak worm error. Since each pixel of the camera with the 135 mm lens subtends 9”, the peak-to-peak worm error is 27 arcsec. This is a well known limitation of the SkyTracker, but it has almost no effect on the image quality with 60 second exposures. The hot pixel track is not straight and uniform, indicating that the mount flexed during the 2 hours.

On the right above is the same 120 images, but this time registered with the 3.5 sigma accumulation option in Sequator. Most of the effect of the hot pixel has disappeared, and there is no indication of the faintest star being effected. If the polar alignment had been near perfect, the hot pixel would have looked like a faint star in an unexpected location. The lesson from this (for me) is that a little polar misalignment can be helpful in eliminating hot pixels and avoiding wasted time to identify "new" supernovae.

There are various algorithms to accomplish this. The algorithm used by Sequator increases the data processing time from 5 minutes to 11 minutes for the 120 images. If anybody knows the details of the Sequator algorithm, please share.

There may be better and faster algorithms for this purpose. Please share your experience with 3 sigma editing with other star registration software.

Cheer,

hha

The first option, of course, is dark subtraction. That will result in less for work for your rejection algorithm. Ideally, the rejection algorithm should mostly be addressing things like cosmic rays and beta decay from radioactive iodine in the cover glass of your camera. In practice, darks aren't perfect--especially with DSLR's and mirrorless system cameras where you can't control temperature. If you aren't using darks, though, you could combine that with your sigma clip combine algorithm so you might not be left with that faint blue trail.

The second tool to use, as mentioned, is dithering. That's not a feature in the Startracker, of course, so there are a couple ways to accomplish it. One of them is slight misalignment of the mount. That has the advantage of not requiring any actions on your part, but it has the disadvantage of slight elongation of your stars since the tracking is off just a bit. It may not be enough to bother you, though, and that's just fine. The one other way you could dither would be to turn the mount off and on for a second between exposures. There are a couple problems wit this--primarily the risk that you will knock the mount slightly such that your frames become significantly misaligned. The second issue is that you would need to babysit the mount for the entire exposure. The third issue is that you have a slight loss of efficiency in your data collection since you would need at least a few seconds between exposures setup in your intervalometer. That's time when you could have been capturing photons. Overall, I think the slight polar misalignment is a better option just because I wouldn't want to stay right next to the camera for every second of a two hour exposure window.

By the way, walking noise may still be present even after the sigma clip stacking. Just as you found it isn't perfect at removing your hot pixel, it won't be perfect in removing walking noise--which is the exact same process, just with smaller variations in pixel brightnesses. With a Startracker, though, there simply is no dither feature, so no way to generate consistently small, random motions between frames. If you push your data hard, though, the walking noise from a polar misalignment will likely become visible.

You mentioned the cost, in imaging efficiency, of using dark frames. It doesn't have to be that bad. You could take your dark frames using just a lens cap if you can take them on a different (cloudy) night with similar temperatures. It's still a hassle, but you don't need to lose imaging time. The real challenge is finding a night with the same temperature as the night you were imaging. Ideally, temps will match within 1*C or so. Obviously, that can be hard to arrange. Worst case, try to make sure your darks are taken on a night with lower temps than your imaging night rather than higher temps--that way your darks may undercorrect slightly, but you don't need to worry about black "pepper" spots showing up in your images from clipping.

There is one last tool I would recommend you take a look at, though I don't know how SEQUATOR handles this... That's building a bad pixel map. Some software can take a stack of dark frames and use those to generate a bad pixel map of all pixels brighter than a particular standard deviation from their neighbors. Then, the software will simply interpolate values for those bad pixels before you register and stack. Check to see if SEQUATOR has this feature. Given the limitations of your Startracker, this may actually be the best of both worlds--most of the improvements from dark frames without the need to have temperature matching darks.

 Jared Willson's gear list:Jared Willson's gear list
Leica Q2 Leica SL2 Hasselblad X1D II 50C Leica SL 90-280mm F2.8–4 Hasselblad XCD 30mm F3.5 +8 more
Alen K
Alen K Senior Member • Posts: 1,298
Re: satellite and hot pixel track suppression in software

hha wrote:

Alen K wrote:

Sequator uses what is often called kappa-sigma clipping or rejection in its "select best pixels" accumulation mode, in three passes according to the manual. The rejection setting (related to standard deviation, I believe) is 2.5 by default. By calling it "accumulation 3.5" I assume you are changing the setting to 3.5. Different images can need different settings but so far 2.5 has always worked for me.

Alen:

Thank you for the detailed response.

The reliable determination of the standard deviation for each pixel is the trickiest part of the clipping algorithm. Pretty reliable with 100 frames, nearly impossible with 10 frames. I select the mildest filter (3.5) because I wanted to avoid clipping out faint star signals, which happens frequently with the 2.5 threshold.

Actually, I checked the limits in the software and it ranges from 1.4 to 3.3 (even in old versions). And I misspoke: the default is 2.0.

As long as I have at least eight images in my stack, I never use anything else. (My largest stack so far was nearly 400 frames.) If I do have less than eight frames, I also select "reduce dynamic noises." If I have less than four frames, which Sequator's manual says is the minimum required for that accumulation mode, I do not activate "select best pixels."

With only eight frames the determination of sigma and what to clip out becomes very hazy. Faint stars disappear. I never use clipping with less than 60 frames.

I haven't actually used it with less than 16 frames, such as with this image. It might have clipped faint stars in that but there are so many of them there that it would be very hard to tell.

And instead of offsetting the polar alignment, I recommend doing random dithering. You may find, as many others have experienced, that PA offset alone will cause walking noise in the background.

The walking noise ( due to gaussian noise spikes) is also virtually eliminated by sigma clipping.

I have my doubts about that, especially if you are limiting the clipping so much. Can you provide a comparison of pure accumulate vs. kappa-sigma clipping that demonstrates that? If you don't feel like proving it to an old doubting Thomas like me, that's okay.

Have you tried the worm error and flexing with your setup?

I don't use a tracker (anymore). My current, highly-portable setup depends on Astrotracer. Random dithering is a natural consequence of how that must be used when creating large stacks (much more work than saner people than I will do). No worm error (no worm gear) and no flexing problems. The camera or lens is mounted directly to a tripod, like this. My other setup uses a Vixen GP-DX Goto mount, so no flexing of that sort even when it is supporting one of my telescopes. Unguided PE is less than 5 arcseconds (when the PPEC is properly trained).

hha1 Junior Member • Posts: 29
Re: satellite and hot pixel track suppression in software

Alen K wrote:

The walking noise ( due to gaussian noise spikes) is also virtually eliminated by sigma clipping.

I have my doubts about that, especially if you are limiting the clipping so much. Can you provide a comparison of pure accumulate vs. kappa-sigma clipping that demonstrates that? If you don't feel like proving it to an old doubting Thomas like me, that's okay.

This is exactly the point I made with the two images  at the start of this post. The one on which shows the hot pixel trail was with simple accumulation, the other shows the result of accumulation with 3.5 sigma clipping. If one assumes that

1. The 100 measurements for each pixel are gaussian distributed with a few positive outliers (noise spikes) , and

2. the standard deviation can be estimated from 100 numbers per pixel (using what is called in the literature a robust standard deviation),

then 3.5 sigma clipping will clip the brightest 0.1% of the pixels  99.9% of the pixels in the image are untouched. With 2 sigma clipping virtually every pixel will have its brightest, but not is darkest  values clipped. This means that the faint stars get fainter. If you are accumulating only 10 frames, than the standard deviation will be very uncertain, as will be the result of the clipping.

Have you tried the worm error and flexing with your setup?

I don't use a tracker (anymore). My current, highly-portable setup depends on Astrotracer. Random dithering is a natural consequence of how that must be used when creating large stacks (much more work than saner people than I will do). No worm error (no worm gear) and no flexing problems. The camera or lens is mounted directly to a tripod, like this. My other setup uses a Vixen GP-DX Goto mount, so no flexing of that sort even when it is supporting one of my telescopes. Unguided PE is less than 5 arcseconds (when the PPEC is properly trained).

Your highly portable setup looks intriguing. Can you share any results with it to win me over?

HHA

Alen K
Alen K Senior Member • Posts: 1,298
Re: satellite and hot pixel track suppression in software

hha1 wrote:

Alen K wrote:

The walking noise ( due to gaussian noise spikes) is also virtually eliminated by sigma clipping.

I have my doubts about that, especially if you are limiting the clipping so much. Can you provide a comparison of pure accumulate vs. kappa-sigma clipping that demonstrates that? If you don't feel like proving it to an old doubting Thomas like me, that's okay.

This is exactly the point I made with the two images at the start of this post. The one on which shows the hot pixel trail was with simple accumulation, the other shows the result of accumulation with 3.5 sigma clipping. If one assumes that

The black level is too low on both images to see if walking noise is a problem on either. I think Jared had a good point about that.

Have you tried the worm error and flexing with your setup?

There is none to correct, on either setup.

I don't use a tracker (anymore). My current, highly-portable setup depends on Astrotracer. Random dithering is a natural consequence of how that must be used when creating large stacks (much more work than saner people than I will do). No worm error (no worm gear) and no flexing problems. The camera or lens is mounted directly to a tripod, like this. My other setup uses a Vixen GP-DX Goto mount, so no flexing of that sort even when it is supporting one of my telescopes. Unguided PE is less than 5 arcseconds (when the PPEC is properly trained).

Your highly portable setup looks intriguing. Can you share any results with it to win me over?

All of the wide-angle, deep-sky images (excepting the ones using film) and the nightscape images in my Astrobin gallery were taken with Astrotracer. All were on the tripod I showed in the other image except for the ones at Grand Canyon and the one I did indoors (!) where I used a smaller travel-friendly tripod. Of course, it doesn't really matter what the camera is mounted to. It just has to be sturdy. Astrotracer will work even if the camera is propped up with a rock.

hha1 Junior Member • Posts: 29
Re: satellite and hot pixel track suppression in software

Jared Willson wrote:

hha wrote:

Airplane, satellite tracks and hot pixels are problem in long time exposure astrophotography. Most cameras have a dark frame subtraction option to eliminate hot pixels, but at the expense of loosing 50% of the exposure time. Most image processing software include an option to suppress artifacts in a long sequence of images. Some have the reputation of being “star eaters”. Here is how I solved this problem with an example using the “Accumulation 3.5” option in SEQUATOR.

I took a 2 hour exposure (120 shots at 60 sec each) using a f=135mm lens at f/4 of the Leo constellation. My Nikon Z6 was mounted on a SkyTracker to eliminate star trails. Rather than waiting for a satellite track, I intentionally misaligned the pole axis by 0.5 degrees. This produces 0.26 degree long star trail in 2 hours. The result of a simple star registration of the 120 images is shown on the left below. This is 1x1 degree cropped from the 10 x 15 degree field, zoomed in at 400% to make the pixels visible.

The stars are perfectly registered. The most prominent star on the lower left is +6.7 (HD97937) in Leo. The galaxy in the upper right is NGC3593 (+11.8). The faintest stars are +15. The bluish trace is produced by a hot pixel. It shows the worm period and a 3 pixel peak-to-peak worm error. Since each pixel of the camera with the 135 mm lens subtends 9”, the peak-to-peak worm error is 27 arcsec. This is a well known limitation of the SkyTracker, but it has almost no effect on the image quality with 60 second exposures. The hot pixel track is not straight and uniform, indicating that the mount flexed during the 2 hours.

On the right above is the same 120 images, but this time registered with the 3.5 sigma accumulation option in Sequator. Most of the effect of the hot pixel has disappeared, and there is no indication of the faintest star being effected. If the polar alignment had been near perfect, the hot pixel would have looked like a faint star in an unexpected location. The lesson from this (for me) is that a little polar misalignment can be helpful in eliminating hot pixels and avoiding wasted time to identify "new" supernovae.

The first option, of course, is dark subtraction.

Do you mean the internal dark subtraction, where ever 30 second exposure is followed by a 30 sec dark frame, the difference is saved as the image? It effectively decreases the SNR achievable with a 60 second exposure by more than a factor 2.

The second tool to use, as mentioned, is dithering. That's not a feature in the Startracker, of course, so there are a couple ways to accomplish it. One of them is slight misalignment of the mount. That has the advantage of not requiring any actions on your part, but it has the disadvantage of slight elongation of your stars since the tracking is off just a bit. It may not be enough to bother you, though, and that's just fine.

This is basically what I did. The 0.5 degree pole misalignment creates an 8 arcsec trail in 60 seconds. My pixels with the 135mm lens are 9 arcsec. The faintest stars with the Z6 cover 2x2 pixels. You have to look very close to see that. I have a program which does that. Indeed, for the 2000 stars identified ui=in the 10x15 degree field with the 135mm lens, the mean star diameter is close to 2 x 3 pixels.

The one other way you could dither would be to turn the mount off and on for a second between exposures. There are a couple problems wit this--primarily the risk that you will knock the mount slightly such that your frames become significantly misaligned. The second issue is that you would need to babysit the mount for the entire exposure. The third issue is that you have a slight loss of efficiency in your data collection since you would need at least a few seconds between exposures setup in your intervalometer. That's time when you could have been capturing photons. Overall, I think the slight polar misalignment is a better option just because I wouldn't want to stay right next to the camera for every second of a two hour exposure window.

You are right. That is the option have use. The camera is controlled with a intervalometer. I set it to take 120 one minute exposures, separated by 2 seconds, Then I can do other things until 2 hours are up.

By the way, walking noise may still be present even after the sigma clip stacking. Just as you found it isn't perfect at removing your hot pixel, it won't be perfect in removing walking noise--which is the exact same process, just with smaller variations in pixel brightnesses. With a Startracker, though, there simply is no dither feature, so no way to generate consistently small, random motions between frames. If you push your data hard, though, the walking noise from a polar misalignment will likely become visible.

I don't recommend an intentional 0.5 degree polar misalignment. I did this  to find out how much effort I need to put into getting a good polar alignment when Polaris is hidden by clouds, trees or mountains. The answer: With a 135mm lens you can get a usable sequence of 60 seconds each, even if you are 0.5 degree misaligned. Good thing to know. With the built in pole axis scope and Polaris visible, alignment within 0.1 degree is easy.

The walking noise is due to the fact that some of the 24 Mpixels are significantly more noisy than others, but it is still gaussian distributed. The probability of a positive 3 sigma or larger excursion is only about 0.2%. The catch is that estimating the robust standard deviation with less than 100 frames becomes increasingly unreliable,

You mentioned the cost, in imaging efficiency, of using dark frames. It doesn't have to be that bad. You could take your dark frames using just a lens cap if you can take them on a different (cloudy) night with similar temperatures. It's still a hassle, but you don't need to lose imaging time.

II like to do it the easy way:  I set the intervalometer to take 120 shots, separated by 2 seconds each, and do other things during the following 2 hours.

The real challenge is finding a night with the same temperature as the night you were imaging. Ideally, temps will match within 1*C or so. Obviously, that can be hard to arrange. Worst case, try to make sure your darks are taken on a night with lower temps than your imaging night rather than higher temps--that way your darks may undercorrect slightly, but you don't need to worry about black "pepper" spots showing up in your images from clipping.

Clipping done properly (by Sequator and others)  does not create black pepper spots, I have a program which creates a list of hot pixels. The list is not very repeatable from night to night. I also tried the "cleaning" function of the Nikon Z6.  Not sure what it does in addition to shaking the sensor, probably creates a map to interpolate across deda pixels.

There is one last tool I would recommend you take a look at, though I don't know how SEQUATOR handles this... That's building a bad pixel map. Some software can take a stack of dark frames and use those to generate a bad pixel map of all pixels brighter than a particular standard deviation from their neighbors. Then, the software will simply interpolate values for those bad pixels before you register and stack. Check to see if SEQUATOR has this feature.

Sequator does not have this feature.

Given the limitations of your Startracker, this may actually be the best of both worlds--most of the improvements from dark frames without the need to have temperature matching darks.

The SkyTracker works well enough for 60 second exposures with a 135 mm lens.  I have used it with a 200 mm lens and the results are good when reasonably pole aligned,

Thanks for your thoughts on this.

HHA

Keyboard shortcuts:
FForum MMy threads