Sharp's new sensor - just curious

Hard to tell from the description, but certainly the pixels here are gigantic, 10 um x 10 um it looks like. And 1% QE, ugh.

There are many ways to make Geiger-mode photodiodes. These are typically avalanche photodiodes that go into a sort of runaway feedback when triggered by a single photoelectron. In the end, the bucket is either empty or fully filled with charge - just 2 states or binary-like. Hence the use of JOT rather than PIXEL.

My paper was motivated by trying to figure out what to do with sub diffraction limit (SDL) pixels. I will try to get it posted as I said. Just havent gotten around to it. Monday perhaps.

-Eric

35mm film was used for cinema projection...

In digital, a dye sub or industrial A4 print at 314ppi is about 3200X2400 or 7.7mp (4:3).
A3 (2xA4 or the centerpage of a magazine) is about 4800x3200 or 15.4mp (3:2).

A sensor with numbers higher than these would hardly show any improvment in resolution within these print sizes, unless we go to 32mp and 64mp respectively, clustering RG2B, so that to avoid most Bayer artifacts (or go for Foveon x3), the Sharp new sensor pitch would allow these and higher numbers in FT and APS.

Of course, when printing larger, say 30"x40", an higher pixel count would show more detail.

When this site started, it was important to realize if resolution was 1 or 2 mp, because it was easy to discern the difference in print; but the higher the count, to see a real difference we have to consider 2x increases and prints over A3.

In film the rule was about 10x for a "sharp" print, with 24x36 just borderline on A3 (10x), with fine grained film, and 6x7 (5x) much neater.

Not necessarely an higher count/smaller photosite is linked to higher noise, and, in studio, flash, tripod, good daylight
ISO 50-200 is enough, with related noise seldom noticeable.

In a way a professional camera (for studio - Hasselblad and so on) can be sold with a performance up to ISO 400, whilst the general public expectations for consumer (or sports) are now ISO 1600-3200, to be used in conditions where an usable image must be obtained without tripod.

Finer or larger grain or pixels are related more to sensitivity than noise, that is mostly an effect of the different clipping level of R-G-B (in b+w the effect is different).

A 12mp sensor has its place in the market next to a 6mp, and with the improved technology its noise might not be noticeably higher, actually in most cases it is lower, so there is no reason why not considering the pixel increase an improvment.
Or smaller pixels allow smaller sensors, for phone etc.

Integrated phones, cameras, video, GPS, MP3s and pocket PC can share the case, battery, charger, LCD, processor and memory, a smaller sensor (and associated lens) is a benefit.

In film, in the late years, most of sales were :
50-100 for landscape, large prints
400 for compacts in covered weather (most prints 6x4)
the grains of these being obviously different.

Looking forward to Mr Fossum' paper on flexible digital film !

I found the pdf file of the gigapixel sensor proposal.

If anyone knows of an easy way to to make it available to others here, let me know and I will email you the paper.

You can write me at fossum at siimpel dot com (sorry, trying to limit my spam)

I will also try to get it posted at Siimpel's site but that means some work by the IT guy.

OK, here is the link

http://www.siimpel.com/Selected%20Publication%20of%20Dr.Fossum.htm

Eric Fossum said:

OK, here is the link
...

Click to expand...

OK - nice proposal.

If this works it will be extremely noise free and without any need for AA filter. The dark part performance will be extremely good. The dynamic range will be impressive. When the exposure gets higher you will get a shoulder due to the fact that the probability for a jot already exposed increases.

In this aproach you will "lose" detection of photons due to the fact that each jot only can be exposed once. In practice this means that you will get the best S/N when approx 50% of the jots are exposed. The performance will get worse for lower and higher exposure.

With a plain photon collector (like todays sensors) the performance goes up until the detector is saturated.

I would guess that, to get good performance, you would need at least 100-1000 more jots than pixels.

--
Roland

Roland Karlsson said:

...

Click to expand...

One problem with this proposal is that, for a certain jot size, the ISO will be fixed, i.e. normal exposure is when 50% of the jots are exposed. So - if you then make an ISO 800 sensor, it IS an ISO 800 sensor.

--
Roland

Roland, I think you missed the grain construct. If a grain consists of 100 jots, then the entire grain is set to exposed if just one jot (or some arbitrary number of jots) is exposed within the grain. Since the grain size can be set in software, the digital film speed (and graininess) can, within limits, be variable. One can determine the digital film speed after the fact depending on where in the camera system the digital development is performed (the jot ORing function).

I think then that even more jots per pixel are required. For example, at 1/10th micron jot pitch, 6400 jots fit in the space of a 8 um x 8 um pixel. Even denser jot pitches would be desirable.

Hope this helps,
Eric

What processing power/memory do you feel is needed at 1/10th micron jots, and in how many (2 or more) steps ?

If I am not wrong the same sensor can be easily used in video mode, being able to sample thousands of times per second the image at a partial resolution.

For what concerns the noise, in digital photo (as GURL said if I am not wrong), what is most evident in a picture is the different treshold of RGB sensors, that show in an image as for instance red spots near black ones, instead of even color. This could be considered in the first processing phase, both in setting (or better offsetting) sensitivity and treshold.

All the other questions might find an easy reply on the first working samples, it is just a matter of manufacturing cost, that could be convevient in just a few years.

Eric Fossum said:

Roland, I think you missed the grain construct. If a grain
consists of 100 jots, then the entire grain is set to exposed if
just one jot (or some arbitrary number of jots) is exposed within
the grain. Since the grain size can be set in software, the
digital film speed (and graininess) can, within limits, be
variable. One can determine the digital film speed after the fact
depending on where in the camera system the digital development is
performed (the jot ORing function).

I think then that even more jots per pixel are required. For
example, at 1/10th micron jot pitch, 6400 jots fit in the space of
a 8 um x 8 um pixel. Even denser jot pitches would be desirable.

Hope this helps,

Click to expand...

OK --- I understand.

But I do not really understand the reason behind the grains. You say that it can be seen as a jot amplifier. Whats wrong with just counting the number of activated jots (e.g. 6400) within a a pixel that is activated and then set the value?

--
Roland

Also the diffraction effects, said to be a limit with smaller pixels, can be minimized by :

sampling a number of images with jolts proportional to airy disks, and recomposing them

if time is available, slightly moving the focus and recomposing the relative images

increasing the "diffraction limit" if needed.

Roland Karlsson said:

OK --- I understand.

But I do not really understand the reason behind the grains. You
say that it can be seen as a jot amplifier. Whats wrong with just
counting the number of activated jots (e.g. 6400) within a a pixel
that is activated and then set the value?

--
Roland

Click to expand...

hi Roland,

I remarked within the paper that even I was unsure if the grain construct added anything. I was trying to emulate film and that is where the idea of a grain (between a jot and a pixel) came to mind. The grain construct adds a layer of non-linear processing (e.g. the OR or threshold function) and frankly I do not know if that helps or hinders the imaging performance. Certainly without the grain non-linear function you are not going to get the D log H characteristic and instead you will get a linear sensor. I just dont know what D log H does to the image quality (other than dynamic range extension). Does it help in noise? Not sure.

Is the extra fine pitch required to do the jot-grain-pixel construct worth the added dynamic range? Don't know that either. Actually, there is little I understand at this point about how well this idea will work. I just think that it is totally do-able and is a great R&D project for either a Micron or some university. If I go back to a university and no one has done this yet, I will work on it for a while. It is easy to simulate and to create synthetic images for noise analysis etc. You dont have to build the chip.

Cheers,
Eric

Eric Fossum said:

I remarked within the paper that even I was unsure if the grain
construct added anything. I was trying to emulate film and that is
where the idea of a grain (between a jot and a pixel) came to mind.
The grain construct adds a layer of non-linear processing (e.g. the
OR or threshold function) and frankly I do not know if that helps
or hinders the imaging performance. Certainly without the grain
non-linear function you are not going to get the D log H
characteristic and instead you will get a linear sensor.

Click to expand...

At very high exposure the sensor will be non linear as the jots will act as very small grains. And if you make the jots larger I don't really see any difference to jots+grains.

Eric Fossum said:

I just
dont know what D log H does to the image quality (other than
dynamic range extension).

Click to expand...

It compresses highlights - which can be done from any HDR image.

Eric Fossum said:

Does it help in noise? Not sure.

Click to expand...

Nope. It limits the memory space used though.

Eric Fossum said:

Is the extra fine pitch required to do the jot-grain-pixel
construct worth the added dynamic range? Don't know that either.
Actually, there is little I understand at this point about how well
this idea will work. I just think that it is totally do-able and
is a great R&D project for either a Micron or some university. If
I go back to a university and no one has done this yet, I will work
on it for a while. It is easy to simulate and to create synthetic
images for noise analysis etc. You dont have to build the chip.

Click to expand...

I was thinking the same myself This is easy stuff to program. And actually - it is easy stuff for doing maths also.

--
Roland

At 0.1µ, and a fast reading time the handshake could be enough to allow several different readings of jots, the different spatial positions during the hand movement would create images that recomposed reduce the diffraction limit.

Fascinating. One of the future avenues of development. Thanks, Guys.

Eric Fossum said:

Roland Karlsson said:

But I do not really understand the reason behind the grains. You
say that it can be seen as a jot amplifier. Whats wrong with just
counting the number of activated jots (e.g. 6400) within a a pixel
that is activated and then set the value?

Click to expand...

Click to expand...

Eric Fossum said:

I remarked within the paper that even I was unsure if the grain
construct added anything. I was trying to emulate film and that is
where the idea of a grain (between a jot and a pixel) came to mind.
The grain construct adds a layer of non-linear processing (e.g. the
OR or threshold function) and frankly I do not know if that helps
or hinders the imaging performance. Certainly without the grain
non-linear function you are not going to get the D log H
characteristic and instead you will get a linear sensor. I just
dont know what D log H does to the image quality (other than
dynamic range extension). Does it help in noise? Not sure.
Is the extra fine pitch required to do the jot-grain-pixel
construct worth the added dynamic range? Don't know that either.

Click to expand...

Eric Fossum said:

Actually, there is little I understand at this point about how well
this idea will work. I just think that it is totally do-able and
is a great R&D project for either a Micron or some university. If
I go back to a university and no one has done this yet, I will work
on it for a while. It is easy to simulate and to create synthetic
images for noise analysis etc. You dont have to build the chip.

Click to expand...

Fascinating idea, Eric. Like Roland, I am not convinced that developing digital grains is a necessary step. Simply counting the exposed jots within a pixel will give a useful S-shaped development characteristic. The binary jots give us more than enough non-linearity.

At low exposures, the count increases linearly with exposure. With perfect binary jots, noise statistics will be identical to an ideal photon counting pixel.

When half the jots are exposed, the incremental gain is reduced by 1/2: only half the original jots are available. There will be a noise penalty compared to an ideal conventional pixel-sized photosensor. On average, each subsequent doubling of exposure halves the number of unexposed jots. Even 1000 jots per pixel should give a huge dynamic range, although a larger number is desirable to reduce noise.

Your proposal for multiple jot read-outs per exposure introduces some interesting design options. We get similar noise statistics and dynamic range for 10000 jots read once, 1000 jots read 10 times, or 100 jots per pixel read 100 times per exposure.

With 10x fewer jots, can we read-out 10x faster? How does this affect the fill-factor or the electrically induced noise?

My initial thought for multiple read-outs was to have a counter (multiple bit storage) for each jot. This might require more silicon for the frame buffer than for the sensor. On reflection, we probably only need a (wider) counter for each colour in each pixel. A larger number of smaller pixels increases our flexibility in trading off noise, dynamic range and resolution. There should be no penalty for binning pixels to improve SNR.

We have the interesting feature that for static subjects we can increase SNR and dynamic range almost indefinitely by increasing the exposure time and number of read-outs, without saturating the sensor.

As you say, calculating noise statistics as a function of jots per pixel and exposure level should be straightforward. Well worth doing a few sums or simulations before considering in detail what the silicon can deliver.

Cheers.
--
Alan Robinson

You are totally correct about getting non-linear characteristics in a "grain-free" sensor, as each jot functions as its own grain. I have always been thinking about much lower exposure levels where the chances of any jot being hit twice was fairly low, compared to hitting a jot within the same grain boundary. But yes, you are correct at higher exposure levels.

I still like the concept of being able to alter grain size for a given image, but again, this can always be done via firmware or software later. Also, I like the flexibility of being to alter the grain shape (e.g. hex-packed) or make the grain centered on the hit jot using region growing.

I would not try to add a counter in each jot. That would be similar to a concept that was proposed at NHK by Fumihiko Ando a long time ago ( 20+ years) and would not allow the use of the tiniest of jots and add the potential for a lot of cross talk as bits are flipping here and there.

Thanks for the discussion. It will take a while for me to remember all the nuances in my mind when I was intensely working on this concept for a few weeks a few years ago. And, when I presented this to the image sensor community, many people were not sure exactly what this was about. It is always fun to present something that challenges the way people think about things. I think this concept is more intuitive to film buffs than to conventional image sensor engineers.

Eric Fossum said:

I would not try to add a counter in each jot. That would be
similar to a concept that was proposed at NHK by Fumihiko Ando a
long time ago ( 20+ years) and would not allow the use of the
tiniest of jots and add the potential for a lot of cross talk as
bits are flipping here and there.

Click to expand...

Agree ... the main advantage of your proposal is its binary nature.

Eric Fossum said:

Thanks for the discussion. It will take a while for me to remember
all the nuances in my mind when I was intensely working on this
concept for a few weeks a few years ago. And, when I presented
this to the image sensor community, many people were not sure
exactly what this was about. It is always fun to present something
that challenges the way people think about things. I think this
concept is more intuitive to film buffs than to conventional image
sensor engineers.

Click to expand...

Hmmm ... although I AM a film buff I am also familiar with digital technology. To use LOTS of binary elements is a common technology in digital electronics. And it is rather straight forward to understand the benefits of your proposal. So ... I wonder what they were struggling with?

--
Roland

Film properties were due in part to the irregular grain size, that allowed to "fill" the image visually, as DR, and sensitivity.

This was more evident in movies, where our eye recorded 2 or 3 images in sequency, which looked sharper than a single one (in a way "layers" in PS).

Regular grain size, up to a certain number of pixels, gives a "firmer" image with perceived higher acutance, but when the information is reduntant (eg over 7.7mp for A4), there is no easily visible advantage.

It is only a matter of processing power/memory cost to drive the jots as needed, 64 bits pcs are becaming common, and the technology can be applied anywhere.

There might be a stage of competition beetween current sized sensors (abt 7mm) with higher pixel count (12mp +) edging in the direction of digital film, and smaller sized sensors mantaining 6 to 10mp, probably glued to the lenses, like the first fixed focal Exilims, for phones etc (liquid zooms ?), then the market will decide.

The new backs at 31mp and 39mp might appear redundant as definition in many cases, but nobody would buy today a 6mp digital back with the same weight/size and a better sensitivity/noise performance by maybe half a stop. 31 to 39mp allow A4 @ 300ppi without bayer artifacts.

And they are manufactured with the old 24x36mm and APS pitch (E-1 to DMR to M8 to 31 and 39mp).

After the start of digital, when bulk size was needed, it is the phone/camera/video technology that is driving the progress in compacts - Dslrs for what concerns the sensor, viceversa digital backs show before the processing/memory/software improvments needed.

Eric Fossum said:

I think then that even more jots per pixel are required. For
example, at 1/10th micron jot pitch, 6400 jots fit in the space of
a 8 um x 8 um pixel. Even denser jot pitches would be desirable.

Click to expand...

So, what 12.5ish bits? What's the advantage over a conventional construction?

--
mumbo jumbo

Roland Karlsson said:

Roland Karlsson said:

I just
dont know what D log H does to the image quality (other than
dynamic range extension).

Click to expand...

It compresses highlights - which can be done from any HDR image.

Click to expand...

Or, more simply, using Fujifilm's dual sensitivity photosites idea for true 'one-shot' behaviour.

--
mumbo jumbo