Awsome write-up - what you say makes sense (although I'm curious as
to why there are more details in highlights - is this a feature of
the sensor?).
(This is a half-explanation because I'm not qualified to fully explain anything about which I'm going to speak.
)
It's not a feature of the sensor so much as it is a feature of physics, or more to the point the human eye. Our eye has two measurements of dynamic range, one relevant to the dynamic range we see every instant and one relevant to the dynamic range over which our eyes can see in different situations.
If you're in a police lineup (hopefully as a paid volunteer) you experience the first kind--the bright lights have to be so many times brighter than the one-way mirror so you can't see the victim through it. This exploits the instantaneous limit of the eye's dynamic range, which is based on the range of your retina.
On the other hand, when you're on a sunny beach, the amount of light bouncing around is something like 12 or 15 orders of magnitude (that's 10^12 to 10^15--I don't remember exactly) brighter than when you're stumbling around your house at night trying to avoid the coffee table. In this case, your eyes have a photosensitive chemical in them that can take the form of two different configurations (called isomers, if you remember your high school chemistry). One isomer of this compound is activated by the presence of light and absorbs light. The other is the stable configuration in the absence of light, and lets light pass. So, in the dark, light passes through your eye to your retina where it is detected mostly by your rods, which is why you don't see much color in a dark situation. When you go outside into full sun, you'll note that your eyes water and often hurt temporarily until they adjust, this is because the light passing through your eye is too great and overstimulates the nerves in your retina to the point of pain. After 2 or 3 minutes, this reversible photosensitive reaction takes over and the light-absorbing isomer of this chemical is formed, and you can see just fine with no pain.
This is a lot of background that basically adds up to one thing: your eyes are logarithmic devices, not linear. If they were linear, there would be no nerve that could possibly handle the range of stimulation that would occur. Even when discussing the instantaneous dynamic range of the eye, it's logarithmically based because your pupil is an aperture and adjusts to allow less or more light by stops (halving or doubling, again not linearly). The only way any device, cameras or eyes, can deal with the vast range of brightnesses out there, is to behave logarithmically.
So if you think about what this means for your camera's sensor, to depict scenes as our eyes see them it has to capture far more detail in the brightest stop of a scene than in the darkest stop. So, half of the available light levels are devoted to this brightest stop, and this is as it should be. It's not a limitation to be overcome--rather, it was designed into the way cameras work in order to depict scenes accurately (meaning, as we see them).
By the way, I should mention that what I've described above is purely constrained to the eye and the eye alone, and has not attempted to address the impact that the brain has on vision. The brain, believe it or not, mucks up the discussion totally, because it adds a lot of processing to the mix that makes a lot of what I said above seem wrong.
For example, you'd be surprised to realize the physical instantaneous dynamic range of the human eye is actually not that great--the 5.5 stops we get from our digital sensors are probably greater. Yet, as photographers it always seems like our cameras never capture enough range to properly depict a scene.
This is because we have an unconscious, persistent memory within the visual cortex. As our pupil runs over an image and relaxes and constricts, it quickly samples a greater dynamic range than the eye is capable of at any one moment. Our brain takes all these samples and merges them together, much like the High Dynamic Range filter in PS CS2. What our eyes see and what we "actually" see are surprisingly very different.
