360Photography

Fossum’s team has created a prototype chip with a variety of pixel designs and readout methods. This included combinations with sufficiently low read noise to allow individual photons to be counted.

The future could include sensors that perfectly describe the light in the scene, that offer new computational possibilities and give film-like latitude in the highlights. And yet we may not ever see them in cameras, says father of the CMOS sensor, Professor Eric Fossum.

We spoke to Fossum shortly after he received, alongside three other pioneers of digital photography, the Queen Elizabeth Prize for Engineering for his work on CMOS sensors. But the topic of our conversation is the future, rather than his past achievements. He now leads a group at the Thayer School of Engineering at Dartmouth, New Hampshire, working on what he calls Quanta Image Sensors (QIS). The team has recently published a paper announcing a breakthrough using the same fabrication process used to make CMOS image sensors.

The perfect sensor?

The principle is to use nanoscale, specialized pixels, called ‘Jots’ to capture light at the level of individual photons. They work in a binary fashion: they’ve either received a photon or they haven’t (as opposed to conventional sensors which accumulate the charge generated by lots of photons during exposure). These jots are read repeatedly to see whether another photon has arrived since they were last checked.

While Fossum is keen to stress that other teams are having some success in the same field (using a slightly different approach), his own team’s work is looking very promising. The paper in the journal Optica shows the team’s technology has been refined such that a 1MJot chip can be read 1000 times per second while still exhibiting sufficiently low read noise that it can distinguish between individual photons.

We can count every photon: you can’t do any better than that

“The Holy Grail is no read noise,” says Fossum: “so that the read signal is proportional to the signal as it arrived.” And the team’s latest paper says they’ve got very close to this, with noise levels so low that the sensor can distinguish between individual photons without getting confused by read noise. This opens up the possibility of cameras that could perfectly describe the light in the scene, even in near total darkness.

A mathematical model showing how noise levels (measured in the root mean square of the number of electrons), affect the ability to interpret small signals. The lower the read noise, the more accurately you can distinguish between individual values in the signal. Diagram from the team’s paper in Optica

Eliminating read noise from the sensor wouldn’t mean totally noiseless photos, since the randomness of the light being captured is a key source of noise, but it’s the best any sensor can possibly achieve. “We can count every photon: you can’t do any better than that,” he says.

The paper, perhaps conservatively, says the technology could be suited to scientific, space, security and low-light imaging applications, but Fossum has clearly also been thinking about conventional photography.

A classic response

“Because it’s binary in nature, its response is comparable to old photographic film,” he says. “In film, when the silver halide was hit by a photon, it’s reduced to a silver atom that isn’t washed away [during processing]. If it’s hit by two photons, it doesn’t make any additional difference.”

This ends up meaning that in bright regions of the image there are ever fewer unexposed silver ions as the exposure goes on. This, in turn makes it less likely that the last few ions will be hit by a photon, so it becomes increasingly difficult to fully saturate the system. The same is true for the tiny, binary Jots: as more of them become saturated, it becomes increasingly difficult to saturate the last few.

“The response is linear at moderate exposure but it trails off to give significant overexposure latitude. It’s a pattern first observed by Hurter and Driffield in 1890,” says Fossum: “they showed the same curve that we measure, experimentally, in our QIS devices.”

Diagram showing the Jots’ exposure response, in comparison to mathematical models of different read noise levels. Note the roll-off at high exposures, comparable to the Hurter Driffield response curves of photographic film. Diagram from the team’s paper in Optica

“That has obvious interest both for still photographers who’re used to shooting film and for cinematographers who’re looking for that kind of response.”

The use of such tiny pixels has other benefits, too: “Jots are below diffraction limits in size. This means the resolution of the system is always higher than the resolution of the lens, which means we never have to worry about aliasing.” While the group’s prototype sensors feature one million Jots, Fossum says their target is one billion.

Beyond conventional photography

Fossum isn’t just thinking about photographic history, though. The tiny size and the approach of repeatedly reading out the sensor challenges the existing concept of single exposures. “At the moment we make motion pictures by shooting a series of snapshots. With QIS it’s more like the reverse process,” he says: constructing still images from precisely captured movement.

Professor Fossum has already been responsible for one revolution in photography: the invention of the CMOS sensor. In December 2017 he was awarded the Queen Elizabeth Prize for Engineering for his work.

Essentially, taking lots of short, sub-frames during an exposure gives you an extra dimension to your images: time. “If you take a single frame, you get a bunch of ones and zeros. If you take another, you quickly build up a cube of ones and zeros,” Fossum says: “For example, if you shoot 100 frames at 1000 frames per second, you get a cube that’s x pixels wide by y pixels tall, but also 100 frames deep.”

This presents some interesting questions, he says: “What do you do with that data? How do you create an image from that very faithful map of where photons arrived?”

“You could choose a number of pixels in x and y but also in the time axis. If you wanted a very sensitive pixel in low light you could combine 10 x 10 Jots in x and y and then maybe combine the data from 100 frames: it’s essentially like increasing the grain size in a more sensitive film.”

Of course you can achieve something comparable to this in conventional digital photography by downscaling an image, but Jots allow greater flexibility, Fossum says: “your pixel size could vary between different parts of the image, so in some places you’d have bigger but more sensitive grains.”

What is the object of photography? Is it artistic or an attempt to perfectly recreate the scene as it was?

The time component also opens up additional possibilities, he says: “if an object moves during these hundred frames, instead of adding all the values from the same location, you could add them at an angle that corresponds to the movement,” so that all the pixels relating to the same object are combined. “We could take out motion blur or remove the scanning effect of a computer screen in video.”

The idea of combining multiple frames in interesting ways is, of course, already becoming a core part of mobile photography, and Fossum says finding all the things that are possible is a challenge he is leaving for others: “From my point of view, we’re building a platform for computational imaging, it’s for others to develop all the ways to use it. A camera would have to take account of the new sensor capabilities.”

But it’ll ask interesting questions, he believes: “What is the object of photography? Is it artistic or an attempt to perfectly recreate the scene as it was? Some of the things we associate with photography are artifacts of the way we capture them.”

Not the only future

With all this going for it, it might seem odd that Fossum isn’t promising to deliver a second revolution in digital imaging. But, having devoted a career to developing technologies and teaching about the challenges, he’s realistic both about the work left to do and the competition any product would face.

“What we’ve already achieved is wonderful. The next challenge is adding color [awareness], but I don’t think that’s going to be particularly problematic. Then there’s power: we’ve shown we can produce a large chip that doesn’t consume or disperse a prohibitively large amount of power. We’re currently at around 27mW but scale it up by 1000 [to get to one billion Jots] and that’s 27W, so we need to cut that by about a factor of ten.”

His concern is more about the current state of the rival technologies: “In order to bring a new technology to replace the existing one, it has to be compellingly better in a number of ways,” he says. “For a few niches, [our technology] is already compelling.” But for photography, the bar is already set very high.

I don’t want our startup to be another esoteric imaging product that fails to find a market

“CMOS technology is pretty awesome right now,” he says, before almost embarrassedly stressing that he’s not claiming the credit for this: “where it is today is the result of the input from thousands of engineers from different companies who’ve contributed towards where we are now.”

Professor Eric Fossum pictured with Dr Jiaju Ma, one of the co-authors of the Optica paper and a co-founder of the spin-off company, Gigajot Technology.

But, for all his cautious words, Fossum is convinced enough by the technology’s potential to have created a company, Gigajot Technology, with his co-researchers. “Finding a sweet spot in the market is a really important part of challenge. It comes back to the things I teach: ‘who is your customer?’ ‘what is your market?’ ‘how are we going to get there?'”

“I don’t want our startup to be another esoteric imaging product that fails to find a market,” he says.

While it’s by no means certain that QIS sensors will make their way into mainstream cameras, it already looks like the technology has tremendous potential for niches such as scientific measurement. This alone shows just how far the technology has come from Fossum’s original idea. As he readily admits: “When we first started this project I wasn’t even sure it could be made to work.”

Articles: Digital Photography Review (dpreview.com)