Computational « 360Photography

RSS

Posts Tagged ‘Computational’

Google’s ex-lead of computational photography Marc Levoy to build new imaging experiences at Adobe

23 Jul

Marc Levoy¹, Google’s former computational photography lead and arguably one of the founding figures of computational approaches to imaging, has joined Adobe as Vice President and Fellow, reporting directly to Chief Technology Officer Abhay Parasnis. At Adobe, Marc will ‘spearhead company-wide technology initiatives focused on computational photography and emerging products, centered on the concept of a universal camera app.’ He will also work closely with Photoshop Camera, Adobe Research, and the machine-learning focused Sensei and Digital Imaging teams.

The imaging sphere was taken by surprise a few months back when Marc left Google where he helped spearhead a revolution in mobile imaging with the excellent success of Pixel phones and their stills and video capabilities. Marc and his colleagues at Google developed HDR+, which uses burst photography alongside clever exposure and merging techniques to increase dynamic range of capture and reduce noise. His work, in conjunction with Peyman Milanfar, also helped Pixel cameras yield visible photos in the dark using Night Sight, and even capture super-resolution data that captured far more detail in ‘zoomed-in’ shots than competitors, despite limited hardware. Google’s burst mode techniques even allowed its cameras to forego traditional demosaicing processes, yielding more detailed images than even competitive cameras with similar sensor sizes.²

Marc Levoy… [is] arguably one of the founding figures of computational approaches to imaging

Marc also championed the use of machine learning to tackle challenges in image capture and processing, leading to better portrait modes, more accurate colors via learning-based white balance, and synthetic re-lighting of faces. Marc helped push the boundaries of what is possible with limited hardware by focusing heavily on the software.

At its core, Adobe is a software company, and so Marc’s expertise is at once relevant. At Adobe, Marc will continue to explore the application of computational photography to Adobe’s imaging and photography products, with one of his focuses being the development of a ‘universal camera app’ that could function across multiple platforms and devices. This should allow Marc to continue his passion for delivering unique and innovative imaging experiences to the masses.

Marc has a knack for distilling complex concepts into simple terms. You can learn about the algorithms and approaches his teams spearheaded in the Pixel phones in our interview above.

More on Marc Levoy

Marc Levoy has a long history of pioneering computational approaches to images, video and computer vision, spanning both industry and academia. He taught at Stanford University, where he remains Professor, Emeritus, and is often credited as popularizing the term ‘computational photography’ through his courses. Before he joined Google he worked as visiting faculty at Google X on the camera for the Explorer Edition of Google Glass. His work early on at Stanford with Google was the basis for Street View in Google Maps. Marc also helped popularize light field photography with his work at Stanford with Mark Horowitz and Pat Hanrahan, advising students like Ren Ng who went on to found Lytro.

Marc also developed his own smartphone apps early on to utilize the potential of burst photography for enhanced image quality with apps like SynthCam. The essential idea – which underpins all multi-imaging techniques today employed by smartphones – is to capture many images to synthesize together into a final image. This technique overcomes the major shortcomings of smartphone cameras: their sensors have such small surface areas and their lenses have such small apertures that the amount of light captured is relatively low. Given that most of the noise in digital images is due to a lack of captured photons (read our primer on the dominant source of noise: shot noise), modern smartphones employ many clever techniques to capture more total light, and in intelligent ways as well to retain both highlight and shadow information while dealing with subject movement from shot to shot. Much of Marc’s early work, as seen in SynthCam, became the basis for the multi-shot noise averaging and bokeh techniques used in Pixel smartphones.

Marc is also passionate about the potential for collaborative efforts and helped develop the ‘Frankencamera’ as an open source platform for experimenting with computational photography. We look forward to the innovation he’ll bring to Adobe, and hope that much of it will be available across platforms and devices to the benefit of photographers at large.

Footnotes:

¹Apart from being well renowned in the fields of imaging and computer graphics, Marc Levoy is himself a photography enthusiast and expert, and while at Stanford taught a Digital Photography class. The course was an in-depth look at everything from sensors to optics to light, color, and image processing, and is available online. We highly recommend our curious readers watch his lectures in video form and also visit Marc’s course website for lecture slides and tools that help you understand the complex concepts both visually and interactively.

²Our own signal:noise ratio analyses of Raw files from the Pixel 4 and representative APS-C and four-thirds cameras show the Pixel 4, in Night Sight mode, to be competitive against both classes of cameras, even slightly out-performing four-thirds cameras (for static scene elements). See our full signal:noise analysis here.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

Computational photography part III: Computational lighting, 3D scene and augmented reality

09 Jun

Editor’s note: This is the third article in a three-part series by guest contributor Vasily Zubarev. The first two parts can be found here:

Part I: What is computational photography?
Part II: Computational sensors and optics

You can visit Vasily’s website where he also demystifies other complex subjects. If you find this article useful we encourage you to give him a small donation so that he can write about other interesting topics.

The article has been lightly edited for clarity and to reflect a handful of industry updates since it first appeared on the author’s own website.

Computational Lighting

Soon we’ll go so goddamn crazy that we’ll want to control the lighting after the photo was taken too. To change the cloudy weather to sunny, or to change the lights on a model’s face after shooting. Now it seems a bit wild, but let’s talk again in ten years.

We’ve already invented a dumb device to control the light — a flash. They have come a long way: from the large lamp boxes that helped avoid the technical limitations of early cameras, to the modern LED flashes that spoil our pictures, so we mainly use them as a flashlight.

Programmable Flash

It’s been a long time since all smartphones switched to Dual LED flashes — a combination of orange and blue LEDs with brightness being adjusted to the color temperature of the shot. In the iPhone, for example, it’s called True Tone and controlled by a small ambient light sensor and a piece of code with a hacky formula.

Link: Demystifying iPhone’s Amber Flashlight

Then we started to think about the problem of all flashes — the overexposed faces and foreground. Everyone did it in their own way. iPhone got Slow Sync Flash, which made the camera increase the shutter speed in the dark. Google Pixel and other Android smartphones started using their depth sensors to combine images with and without flash, quickly made one by one. The foreground was taken from the photo with the flash while the background remained lit by ambient illumination.

The further use of a programmable multi-flash is vague. The only interesting application was found in computer vision, where it was used once in assembly schemes (like for Ikea book shelves) to detect the borders of objects more accurately. See the article below.

Link: Non-photorealistic Camera: Depth Edge Detection and Stylized Rendering using Multi-Flash Imaging

Lightstage

Light is fast. It’s always made light coding an easy thing to do. We can change the lighting a hundred times per shot and still not get close to its speed. That’s how Lighstage was created back in 2005.

Video link: Lighstage demo video

The essence of the method is to highlight the object from all possible angles in each shot of a real 24 fps movie. To get this done, we use 150+ lamps and a high-speed camera that captures hundreds of shots with different lighting conditions per shot.

A similar approach is now used when shooting mixed CGI graphics in movies. It allows you to fully control the lighting of the object in post-production, placing it in scenes with absolutely random lighting. We just grab the shots illuminated from the required angle, tint them a little, done.

Unfortunately, it’s hard to do it on mobile devices, but probably someone will like the idea and execute it. I’ve seen an app from guys who shot a 3D face model, illuminating it with the phone flashlight from different sides.

Lidar and Time-of-Flight Camera

Lidar is a device that determines the distance to the object. Thanks to a recent hype of self-driving cars, now we can find a cheap lidar in any dumpster. You’ve probably seen these rotating thingys on the roof of some vehicles? These are lidars.

We still can’t fit a laser lidar into a smartphone, but we can go with its younger brother — time-of-flight camera. The idea is ridiculously simple — a special separate camera with an LED-flash above it. The camera measures how quickly the light reaches the objects and creates a depth map of the scene.

The accuracy of modern ToF cameras is about a centimeter. The latest Samsung and Huawei top models use them to create a bokeh map and for better autofocus in the dark. The latter, by the way, is quite good. I wish every device had one.

Knowing the exact depth of field will be useful in the coming era of augmented reality. It will be much more accurate and effortless to shoot at the surfaces with lidar to make the first mapping in 3D than analyzing camera images.

Projector Illumination

To finally get serious about computational lighting, we have to switch from regular LED flashes to projectors — devices that can project a 2D picture on a surface. Even a simple monochrome grid will be a good start for smartphones.

The first benefit of the projector is that it can illuminate only the part of the image that needs to be illuminated. No more burnt faces in the foreground. Objects can be recognized and ignored, just like laser headlights of some modern cars don’t blind the oncoming drivers but illuminate pedestrians. Even with the minimum resolution of the projector, such as 100×100 dots, the possibilities are exciting.

Today, you can’t surprise a kid with a car with a controllable light.

The second and more realistic use of the projector is to project an invisible grid on a scene to build a depth map. With a grid like this, you can safely throw away all your neural networks and lidars. All the distances to the objects in the image now can be calculated with the simplest computer vision algorithms. It was done in Microsoft Kinect times (rest in peace), and it was great.

Of course, it’s worth remembering here the Dot Projector for Face ID on iPhone X and above. That’s our first small step towards projector technology, but quite a noticeable one.

Dot Projector in iPhone X.

Vasily Zubarev is a Berlin-based Python developer and a hobbyist photographer and blogger. To see more of his work, visit his website or follow him on Instagram and Twitter.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

Computational photography part II: Computational sensors and optics

08 Jun

Editor’s note: This is the second article in a three-part series by guest contributor Vasily Zubarev. The first and third parts can be found here:

Part I: What is computational photography?
Part III: Computational lighting, 3D scene and augmented reality (coming soon)

Computational Sensor: Plenoptic and Light Fields

Well, our sensors are crap. We simply got used to it and trying to do our best with them. They haven’t changed much in their design from the beginning of time. Technical process was the only thing that improved — we reduced the distance between pixels, fought read noise, increased readout speeds and added specific pixels for phase-detection autofocus systems. But even if we take the most expensive camera to try to photograph a running cat in the indoor light, the cat will win.

Video link: The Science of Camera Sensors

We’ve been trying to invent a better sensor for a long time. You can google a lot of research in this field by “computational sensor” or “non-Bayer sensor” queries. Even the Pixel Shifting example can be referred to as an attempt to improve sensors with calculations.

The most promising stories of the last twenty years, though, come to us from plenoptic cameras.

To calm your sense of impending boring math, I’ll throw in the insider’s note — the last Google Pixel camera is a little bit plenoptic. With only two pixels in one, there’s still enough to calculate a fair optical depth of field map without having a second camera like everyone else.

Plenoptics is a powerful weapon that hasn’t fired yet.

Plenoptic Camera

Invented in 1994. For the first time assembled at Stanford in 2004. The first consumer product — Lytro, released in 2012. The VR industry is now actively experimenting with similar technologies.

Plenoptic camera differs from the normal one by only one modification. Its sensor is covered with a grid of lenses, each of which covers several real pixels. Something like this:

If we place the grid and sensor at the right distance, we’ll see sharp pixel clusters containing mini-versions of the original image on the final RAW image.

Video link: Muted video showing RAW editing process

Apparently, if you take only one central pixel from each cluster and build the image only from them, it won’t be any different from one taken with a standard camera. Yes, we lose a bit in resolution, but we’ll just ask Sony to stuff more megapixels in the next sensor.

That’s where the fun part begins. If you take another pixel from each cluster and build the image again, you again get a standard photo, only as if it was taken with a camera shifted by one pixel in space. Thus, with 10×10 pixel clusters, we get 100 images from “slightly” different angles.

The more the cluster size, the more images we have. Resolution is lower, though. In the world of smartphones with 41-megapixel sensors, everything has a limit, although we can neglect resolution a bit. We have to keep the balance.

Link: plenoptic.info – about plenoptics, with python code samples

Alright, we’ve got a plenoptic camera. What can we do with it?

Fair refocusing

The feature that everyone was buzzing about in the articles covering Lytro is the possibility to adjust focus after the shot was taken. “Fair” means we don’t use any deblurring algorithms, but rather only available pixels, picking or averaging in the right order.

A RAW photo taken with a plenoptic camera looks weird. To get the usual sharp JPEG out of it, you have to assemble it first. The result will vary depending on how we select the pixels from the RAW.

The farther the cluster is from the point of impact of the original ray, the more defocused the ray is. Because the optics. To get the image shifted in focus, we only need to choose the pixels at the desired distance from the original — either closer or farther.

The picture should be read from right to left as we are sort of restoring the image, knowing the pixels on the sensor. We get a sharp original image on top, and below we calculate what was behind it. That is, we shift the focus computationally.

The process of shifting the focus forward is a bit more complicated as we have fewer pixels in these parts of the clusters. In the beginning, Lytro developers didn’t even want to let the user focus manually because of that — the camera made a decision itself using the software. Users didn’t like that, so the feature was added in the late versions as “creative mode”, but with very limited refocus for exactly that reason.

Depth Map and 3D using a single lens

One of the simplest operations in plenoptics is to get a depth map. You just need to gather two different images and calculate how the objects are shifted between them. The more the shift — the farther away from the camera the object is.

Google recently bought and killed Lytro, but used their technology for its VR and… Pixel’s camera. Starting with the Pixel 2, the camera became “a little bit” plenoptic, though with only two pixels per cluster. As a result, Google doesn’t need to install a second camera like all the other cool kids. Instead, they can calculate a depth map from one photo.

Images which top and bottom subpixels of the Google Pixel camera see. The right one is animated for clarity (click to enlarge and see animation). Source: Google

The depth map is additionally processed with neural networks to make the background blur more even. Source: Google

Link: Portrait mode on the Pixel 2 and Pixel 2 XL smartphones

The depth map is built on two shots shifted by one sub-pixel. This is enough to calculate a rudimentary depth map and separate the foreground from the background to blur it out with some fashionable bokeh. The result of this stratification is still smoothed and “improved” by neural networks which are trained to improve depth maps (rather than to observe, as many people think).

The trick is that we got plenoptics in smartphones almost at no charge. We already put lenses on these tiny sensors to increase the luminous flux at least somehow. Some patents from Google suggest that future Pixel phones may go further and cover four photodiodes with a lens.

Slicing layers and objects

You don’t see your nose because your brain combines a final image from both of your eyes. Close one eye, and you will see a huge Egyptian pyramid at the edge.

The same effect can be achieved in a plenoptic camera. By assembling shifted images from pixels of different clusters, we can look at the object as if from several points. Same as our eyes do. It gives us two cool opportunities. First is we can estimate the approximate distance to the objects, which allows us easily separate the foreground from the background as in life. And second, if the object is small, we can completely remove it from the photo since we can effectively look around the object. Like a nose. Just clone it out. Optically, for real, with no photoshop.

Using this, we can cut out trees between the camera and the object or remove the falling confetti, as in the video below.

“Optical” stabilization with no optics

From a plenoptic RAW, you can make a hundred of photos with several pixels shift over the entire sensor area. Accordingly, we have a tube of lens diameter within which we can move the shooting point freely, thereby offsetting the shake of the image.

Technically, stabilization is still optical, because we don’t have to calculate anything — we just select pixels in the right places. On the other hand, any plenoptic camera sacrifices the number of megapixels in favor of plenoptic capabilities, and any digital stabilizer works the same way. It’s nice to have it as a bonus, but using it only for its sake is costly.

The larger the sensor and lens, the bigger window for movement. The more camera capabilities, the more ozone holes from supplying this circus with electricity and cooling. Yeah, technology!

Fighting with Bayer filter

Bayer filter is still necessary even with a plenoptic camera. We haven’t come up with any other way of getting a colorful digital image. And using a plenoptic RAW, we can average the color not only by the group of nearby pixels, as in classic demosaicing, but also using dozens of its copies in neighboring clusters.

It’s called “computable super-resolution” in some articles, but I would question it. In fact, we reduce the real resolution of the sensor in these some dozen times first in order to proudly restore it again. You have to try hard to sell it to someone.

But technically it’s still more interesting than shaking the sensor in a pixel shifting spasm.

Computational aperture (bokeh)

Those who like to shoot bokeh hearts will be thrilled. Since we know how to control the refocus, we can move on and take only a few pixels from the unfocused image and others from the normal one. Thus we can get an aperture of any shape. Yay! (No)

Many more tricks for video

So, not to move too far away from the photo topic, everyone who’s interested should check out the links above and below. They contain about half a dozen other interesting applications of a plenoptic camera.

Video link: Watch Lytro Change Cinematography Forever

Light Field: More than a photo, less than VR

Usually, the explanation of plenoptics starts with light fields. And yes, from the science perspective, the plenoptic camera captures the light field, not just the photo. Plenus comes from the Latin “full”, i.e., collecting all the information about the rays of light. Just like a Parliament plenary session.

Let’s get to the bottom of this to understand what a light field is and why we need it.

Traditional photos are two-dimensional. When a ray hits a sensor there will be a corresponding pixel in the photo that records simply its intensity. The camera doesn’t care where the ray came from, whether it accidentally fell from aside or was reflected off of another object. The photo captures only the point of intersection of the ray with the surface of the sensor. So it’s kinda 2D.

Light field images are similar, but with a new component — the origin and angle of each ray. The microlens array in front of the sensor is calibrated such that each lens samples a certain portion of the aperture of the main lens, and each pixel behind each lens samples a certain set of ray angles. And since light rays emanating from an object with different angles fall across different pixels on a light field camera’s sensor, you can build an understanding of all the different incoming angles of light rays from this object. This means the camera effectively captures the ray vectors in 3D space. Like calculating the lighting of a video game, but the other way around — we’re trying to catch the scene, not create it. The light field is the set of all the light rays in our scene — capturing both the intensity and angular information about each ray.

There are a lot of mathematical models of light fields. Here’s one of the most representative.

The light field is essentially a visual model of the space around it. We can easily compute any photo within this space mathematically. Point of view, depth of field, aperture — all these are also computable; however, one can only reposition the point of view so much, determined by the entrance pupil of the main lens. That is, the amount of freedom with which you can change the field of view depends upon the breadth of perspectives you’ve captured, which is necessarily limited.

I love to draw an analogy with a city here. Photography is like your favorite path from your home to the bar you always remember, while the light field is a map of the whole town. Using the map, you can calculate any route from point A to B. In the same way, knowing the light field, we can calculate any photo.

For an ordinary photo it’s overkill, I agree. But here comes VR, where light fields are one of the most promising areas of development.

Having a light field model of an object or a room allows you to see this object or a room from multiple perspectives, with motion parallax and other depth cues like realistic changes in textures and lighting as you move your head. You can even travel through a space, albeit to a limited degree. It feels like virtual reality, but it’s no longer necessary to build a 3D-model of the room. We can ‘simply’ capture all the rays inside it and calculate many different pictures from within that volume. Simply, yeah. That’s what we’re fighting over.

Link: Google AR and VR: Experimenting with Light Fields

Vasily Zubarev is a Berlin-based Python developer and a hobbyist photographer and blogger. To see more of his work, visit his website or follow him on Instagram and Twitter.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

Computational photography part I: What is computational photography?

04 Jun

Editor’s note: The term ‘computational photography’ gets used a lot these days, but what exactly does it mean? In this article, the first in a three-part series, guest contributor Vasily Zubarev takes us on a journey from present to future, explaining computational photography today, where it’s going and how it will change the very essence of photography.

Series overview:

Part I: What is Computational Photography?
Part II: Computational sensors and optics (coming soon)
Part III: Computational lighting, 3D scene and augmented reality (coming soon)

Computational Photography: From Selfies to Black Holes

It’s impossible to imagine a smartphone presentation today without dancing around its camera. Google makes Pixel shoot in the dark, Huawei zooms like a telescope, Samsung puts lidars inside, and Apple presents the new world’s roundest corners. Illegal level of innovations happening here.

DSLRs, on the other hand, seem half dead. Sony showers everybody with a new sensor-megapixel rain every year, while manufacturers lazily update the minor version number and keep lying on piles of cash from movie makers. I have a $ 3000 Nikon on my desk, but I take an iPhone on my travels. Why?

It’s impossible to imagine a smartphone presentation today without dancing around its camera.

I went online with this question. There, I saw a lot of debate about “algorithms” and “neural networks”, though no one could explain how exactly they affect a photo. Journalists are loudly reading the number of megapixels from press releases, bloggers are shutting down the Internet with more unboxings, and the camera-nerds are overflowing it with “sensual perception of the sensor color palette”. Ah, Internet. You gave us access to all the information. Love you.

Thus, I spent half of my life to understand the whole thing on my own. I’ll try to explain everything I found in this article, otherwise I’ll forget it in a month.

What is Computational Photography?

Everywhere, including Wikipedia, you get a definition like this: computational photography is a digital image capture and processing techniques that use digital computation instead of optical processes. Everything is fine with it except that it’s bullshit. The fuzziness of the official definitions kinda indicates that we still have no idea what are we doing.

Stanford Professor and pioneer of computational photography Marc Levoy (he was also behind many of the innovations in Google’s Pixel cameras) gives another definition – computational imaging techniques enhance or extend the capabilities of digital photography in which the output is an ordinary photograph, but one that could not have been taken by a traditional camera. I like it more, and in the article, I will follow this definition.

So, the smartphones were to blame for everything.

So, the smartphones were to blame for everything. Smartphones had no choice but to give life to a new kind of photography — computational.

They had little noisy sensors and tiny slow lenses. According to all the laws of physics, they could only bring us pain and suffering. And they did. Until some devs figured out how to use their strengths to overcome the weaknesses: fast electronic shutters, powerful processors, and software.

Most of the significant research in the computational photography field was done in 2005-2015, which counts as yesterday in science. That means, right now, just in front of our eyes and inside our pockets, there’s a new field of knowledge and technology rising that never existed before.

Computational photography isn’t just about the bokeh on selfies. A recent photograph of a black hole would not have been taken without using computational photography methods. To take such picture with a standard telescope, we would have to make it the size of the Earth. However, by combining the data of eight radio telescopes at different locations of our Earth-ball and writing some cool Python scripts, we got the world’s first picture of the event horizon.

It’s still good for selfies though, don’t worry.

Link: Computational Photography: Principles and Practice
Link: Marc Levoy: New Techniques in Computational photography

(I’m going to insert such links in the course of the story. They will lead you to the rare brilliant articles or videos that I found, and allow you to dive deeper into a topic if you suddenly become interested. Because I physically can’t tell you everything in one article.)

The Beginning: Digital Processing

Let’s get back to 2010. Justin Bieber released his first album and the Burj Khalifa had just opened in Dubai, but we couldn’t even capture these two great universe events because our photos were noisy 2-megapixel JPEGs. We got the first irresistible desire to hide the worthlessness of mobile cameras by using “vintage” presets. Instagram comes out.

Math and Instagram

With the release of Instagram, everyone got obsessed with filters. As the man who reverse engineered the X-Pro II, Lo-Fi, and Valencia for, of course, research (hehe) purposes, I still remember that they comprised three components:

Color settings (Hue, Saturation, Lightness, Contrast, Levels, etc.) are simple coefficients, just like in any presets that photographers used since ancient times.

Tone Mapping is a vector of values, each tells us that “red with a hue of 128 should be turned into a hue of 240”. It’s often represented as a single-pixel picture, like this one. This is an example for the X-Pro II filter.

Overlay — translucent picture with dust, grain, vignette, and everything else that can be applied from above to get the (not at all, yeah) banal effect of the old film. Used rarely.

Modern filters have not gone far from these three, but have become a little more complicated from the math perspective. With the advent of hardware shaders and OpenCL on smartphones, they were quickly rewritten under the GPU, and it was considered insanely cool. For 2012, of course. Today any kid can do the same thing on CSS, but he still won’t invite a girl to prom.

However, progress in the area of filters has not stopped there. Guys from Dehan?er, for example, are getting very hands-on with non-linear filters. Instead of poor-human tone-mapping, they use more posh and complex non-linear transformations, which opens up many more opportunities, according to them.

With the release of Instagram, everyone got obsessed with filters.

You can do a lot of things with non-linear transformations, but they are incredibly complex, and we humans are incredibly stupid. As soon as it comes to non-linear transformations, we prefer to go with numerical methods or run neural networks to do our job. The same thing happens here.

Automation and Dreams of a “Masterpiece” Button

When everybody got used to filters, we started to integrate them right into our cameras. It’s hidden in history whoever was the first manufacturer to implement this, but just to understand how long ago it was, think, that in iOS 5.0 released in 2011 we already had a public API for Auto Enhancing Images. Only Steve Jobs knows how long it was in use before it opened to the public.

The automation was doing the same thing that any of us does by opening the photo editor — it fixed the lights and shadows, increased the brightness, took away the red eyes, and fixed the face color. Users didn’t even know that “dramatically improved camera” was just the merit of a couple of new lines of code.

ML Enhance in Pixelmator.

Today, the battles for the Masterpiece button have moved to the machine learning field. Tired of playing with tone-mapping, everyone rushed to the hype train CNN’s and GAN’s and started forcing computers to move the sliders for us. In other words, to use an input image to determine a set of optimal parameters that will bring the given image closer to a particular subjective understanding of “good photography”. Check out how it’s implemented in Pixelmator Pro and other editors who’s luring you with their fancy “ML” features stated on a landing page. It doesn’t always work well, as you can guess. But you can always take the datasets and train your own network to beat these guys, using the links below. Or not.

Link: Image Enhancement Papers
Link: DSLR-Quality Photos on Mobile Devices with Deep Convolutional Networks

Vasily Zubarev is a Berlin-based Python developer and a hobbyist photographer and blogger. To see more of his work, visit his website or follow him on Instagram and Twitter.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

Lead engineer for computational imaging on Pixel devices leaves Google

14 May

According to a report by industry publication The Information, two key executives in Google’s Pixel team left the company earlier this year. One of them is the former General Manager of the Pixel Smartphones Business Unit, Mario Queiroz. According to his Linkedin profile, he left Google at the end of January to take on the role of Executive Vice President at data security company Palo Alto Networks.

A few months earlier, and two months before the launch of the Pixel 4 devices in October 2019, he had already moved internally from the Pixel team into a role that directly reported to Google CEO Sundar Pichai.

From an imaging point of view, the second executive leaving the Pixel team and company is more interesting, though: Marc Levoy has been a Computer Science professor at Stanford University since 1990 and since 2014, in his role as Distinguished Engineer at Google, had been leading the Pixel team that developed computational photography technologies for Pixel smartphones, including HDR+, Portrait Mode, and Night Sight.

Since its inception the Pixel smartphones series had excelled in the camera department, receiving positive camera reviews across the board. With the Pixel phones using very similar camera hardware to its direct rivals, a lot of the Pixel’s camera success could likely be attributed to the innovative imaging software features mentioned above.

However, things look slightly different for the latest Pixel 4 generation that was launched in October 2019. While many of the software features and functions were updated and improved, the camera hardware looks a little old next to the top-end competition. Companies like Samsung, Huawei and Xiaomi offer larger sensors with higher resolutions and longer tele lenses, and combine those hardware features with computational imaging methods, achieving excellent results. The Pixel 4 is also one of very few high-end phones to not feature a dedicated ultra-wide camera.

The Pixel 4 camera is still excellent in many situations but it’s hard to argue that Google has, at least do a degree, lost the leadership role in mobile imaging that it had established with previous Pixel phone generations.

It looks like internally there has been some discontent with other aspects of the Pixel 4 hardware, too. The report from The Information also details some criticism from Google hardware lead Rick Osterloh on the Pixel 4 battery:

At a hardware team all-hands meeting in the fall, ahead of the October launch in New York, Osterloh informed staff about his own misgivings. He told them he did not agree with some of the decisions made about the phone, according to two people who were present at the meeting. In particular, he was disappointed in its battery power.

Battery and camera performance are likely only two out of a range of factors that caused Pixel 4 sales figures to decrease when compared to its predecessors. IDC estimates that Google shipped around 2 million Pixel 4 units in the first two quarters the phone was on sale, compared to 3.5 million Pixel 3 units and almost 3 million Pixel 3A devices.

These figures are also relatively small when compared to the largest competitors. According to IDC Apple sold a whopping 73.8 million iPhones in the fourth quarter of 2019, for example.

It’s not entirely clear, but likely, that the departures of Queiroz and Levoy are linked to the Pixel 4’s performance in the marketplace. What will it mean for future Pixel phones and their cameras? We will only know once we hold the Pixel 5 in our hands but we hope Google will continue to surprise us with new and innovative technologies that get the most out of smartphone cameras.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

Emojivision uses computational photography to turn your photos into emojis

04 Sep

A free new camera app for iOS called Emojivision allows you to capture images composed entirely of emoji. The app was created by Gabriel O’Flaherty-Chan, according to TechCrunch, which reports that Emojivision uses computational photography to break an image down into its core color palette, then rebuilds it using similarly colored emoji in near-real-time.

The app can be used to take any image, as with the native camera app, and also to apply the emoji filters to existing images located in the phone’s camera roll. The app is free, but enthusiastic users can pay $ 2.79 USD to get additional emoji packs. For developers, the Emojivision project is located with technical details on GitHub.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

Leica will cut 100 jobs HQ, add 40 ‘digital experts’ to push its computational imaging forward

22 Jun

German business newspaper Handelsblatt is reporting [translated to English] that, as part of a restructuring process, Leica will eliminate up to 100 jobs at the company’s headquarters and add up to 40 new ‘digital experts’ to push forward its smartphone and computational photography technology.

According to Handelsblatt, Leica’s restructuring is due to ‘profound changes in the market.’ Leica CEO, Matthias Harsch, is quoted as saying ‘We are facing the second digital revolution in the camera business,’ an obvious nod to the market’s movement away from dedicated cameras to smartphone cameras.

Leica CEO Matthias Harsch

In addition to the recent controversy surrounding the ‘Tank Man’ advertisement that caused an uproar last month, Handelsblatt also notes Leica’s partnership with Chinese smartphone manufacturer Huawei as a potential cause of worry at Leica. Huawei, who has partnered with Leica to put its camera technology in Huawei devices, is reported by Reuters to have lost its licensing of Google’s Android operating system as a part of restrictions put in place by the U.S. government amidst security concerns of Huawei devices.

Despite the uncertainty of Huawei’s future, Harsch sounds confident the partnership will remain beneficial and further states the significant role smartphone photography will play in Leica’s business going forward, saying:

‘The camera function with smartphones is a core business of our future […] After all, thanks to their smartphones, people have never photographed as much as they do today.’

An illustration of the Leica triple-camera system inside Huawei’s P30 smartphone.

Harsch also specifically notes the growing role of artificial intelligence and computational photography in digital images. Leica has been working alongside Huawei for the past four years, developing both the hardware and—arguably more importantly—the software used for mobile image capture and processing. He says ‘These experiences [developing smartphone cameras technology] can be used for the further development of our classic cameras.’

Evidence of Leica’s interest in becoming a leader in computational photography is backed by the news that it will be hiring up to 40 experts in the field, investing a ‘double-digit million amount,’ according to Handelsblatt.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

Five ways Google Pixel 3 pushes the boundaries of computational photography

11 Oct

With the launch of the Google Pixel 3, smartphone cameras have taken yet another leap in capability. I had the opportunity to sit down with Isaac Reynolds, Product Manager for Camera on Pixel, and Marc Levoy, Distinguished Engineer and Computational Photography Lead at Google, to learn more about the technology behind the new camera in the Pixel 3.

One of the first things you might notice about the Pixel 3 is the single rear camera. At a time when we’re seeing companies add dual, triple, even quad-camera setups, one main camera seems at first an odd choice.

But after speaking to Marc and Isaac I think that the Pixel camera team is taking the correct approach – at least for now. Any technology that makes a single camera better will make multiple cameras in future models that much better, and we’ve seen in the past that a single camera approach can outperform a dual camera approach in Portrait Mode, particularly when the telephoto camera module has a smaller sensor and slower lens, or lacks reliable autofocus.

Let’s take a closer look at some of the Pixel 3’s core technologies.

1. Super Res Zoom

Last year the Pixel 2 showed us what was possible with burst photography. HDR+ was its secret sauce, and it worked by constantly buffering nine frames in memory. When you press the shutter, the camera essentially goes back in time to those last nine frames¹, breaks each of them up into thousands of ’tiles’, aligns them all, and then averages them.

Breaking each image into small tiles allows for advanced alignment even when the photographer or subject introduces movement. Blurred elements in some shots can be discarded, or subjects that have moved from frame to frame can be realigned. Averaging simulates the effects of shooting with a larger sensor by ‘evening out’ noise. And going back in time to the last 9 frames captured right before you hit the shutter button means there’s zero shutter lag.

Like the Pixel 2, HDR+ allows the Pixel 3 to render sharp, low noise images even in high contrast situations. Click image to view the level of detail at 100%. Photo: Google

This year, the Pixel 3 pushes all this further. It uses HDR+ burst photography to buffer up to 15 images², and then employs super-resolution techniques to increase the resolution of the image beyond what the sensor and lens combination would traditionally achieve³. Subtle shifts from handheld shake and optical image stabilization (OIS) allow scene detail to be localized with sub-pixel precision, since shifts are unlikely to be exact multiples of a pixel.

In fact, I was told the shifts are carefully controlled by the optical image stabilization system. “We can demonstrate the way the optical image stabilization moves very slightly” remarked Marc Levoy. Precise sub-pixel shifts are not necessary at the sensor level though; instead, OIS is used to uniformly distribute a bunch of scene samples across a pixel, and then the images are aligned to sub-pixel precision in software.

We get a red, green, and blue filter behind every pixel just because of the way we shake the lens, so there’s no more need to demosaic

But Google – and Peyman Milanfar’s research team working on this particular feature – didn’t stop there. “We get a red, green, and blue filter behind every pixel just because of the way we shake the lens, so there’s no more need to demosaic” explains Marc. If you have enough samples, you can expect any scene element to have fallen on a red, green, and blue pixel. After alignment, then, you have R, G, and B information for any given scene element, which removes the need to demosaic. That itself leads to an increase in resolution (since you don’t have to interpolate spatial data from neighboring pixels), and a decrease in noise since the math required for demosaicing is itself a source of noise. The benefits are essentially similar to what you get when shooting pixel shift modes on dedicated cameras.


Normal wide-angle (28mm equiv.)	Super Res Zoom

There’s a small catch to all this – at least for now. Super Res only activates at 1.2x zoom or more. Not in the default ‘zoomed out’ 28mm equivalent mode. As expected, the lower your level of zoom, the more impressed you’ll be with the resulting Super Res images, and naturally the resolving power of the lens will be a limitation. But the claim is that you can get “digital zoom roughly competitive with a 2x optical zoom” according to Isaac Reynolds, and it all happens right on the phone.

The results I was shown at Google appeared to be more impressive than the example we were provided above, no doubt at least in part due to the extreme zoom of our example here. We’ll reserve judgement until we’ve had a chance to test the feature for ourselves.

Would the Pixel 3 benefit from a second rear camera? For certain scenarios – still landscapes for example – probably. But having more cameras doesn’t always mean better capabilities. Quite often ‘second’ cameras have worse low light performance due to a smaller sensor and slower lens, as well as poor autofocus due to the lack of, or fewer, phase-detect pixels. One huge advantage of Pixel’s Portrait Mode is that its autofocus doesn’t differ from normal wide-angle shooting: dual pixel AF combined with HDR+ and pixel-binning yields incredible low light performance, even with fast moving erratic subjects.

2. Computational Raw

The Pixel 3 introduces ‘computational Raw’ capture in the default camera app. Isaac stressed that when Google decided to enable Raw in its Pixel cameras, they wanted to do it right, taking advantage of the phone’s computational power.

Our Raw file is the result of aligning and merging multiple frames, which makes it look more like the result of a DSLR

“There’s one key difference relative to the rest of the industry. Our DNG is the result of aligning and merging [up to 15] multiple frames… which makes it look more like the result of a DSLR” explains Marc. There’s no exaggeration here: we know very well that image quality tends to scale with sensor size thanks to a greater amount of total light collected per exposure, which reduces the impact of the most dominant source of noise in images: photon shot, or statistical, noise.

The Pixel cameras can effectively make up for their small sensor sizes by capturing more total light through multiple exposures, while aligning moving objects from frame to frame so they can still be averaged to decrease noise. That means better low light performance and higher dynamic range than what you’d expect from such a small sensor.

Shooting Raw allows you to take advantage of that extra range: by pulling back blown highlights and raising shadows otherwise clipped to black in the JPEG, and with full freedom over white balance in post thanks to the fact that there’s no scaling of the color channels before the Raw file is written.

Pixel 3 introduces in-camera computational Raw capture.

Such ‘merged’ Raw files represent a major threat to traditional cameras. The math alone suggests that, solely based on sensor size, 15 averaged frames from the Pixel 3 sensor should compete with APS-C sized sensors in terms of noise levels. There are more factors at play, including fill factor, quantum efficiency and microlens design, but needless to say we’re very excited to get the Pixel 3 into our studio scene and compare it with dedicated cameras in Raw mode, where the effects of the JPEG engine can be decoupled from raw performance.

While solutions do exist for combining multiple Raws from traditional cameras with alignment into a single output DNG, having an integrated solution in a smartphone that takes advantage of Google’s frankly class-leading tile-based align and merge – with no ghosting artifacts even with moving objects in the frame – is incredibly exciting. This feature should prove highly beneficial to enthusiast photographers. And what’s more – Raws are automatically uploaded to Google Photos, so you don’t have to worry about transferring them as you do with traditional cameras.

3. Synthetic Fill Flash

‘Synthetic Fill Flash’ adds a glow to human subjects, as if a reflector were held out in front of them. Photo: Google

Often a photographer will use a reflector to light the faces of backlit subjects. Pixel 3 does this computationally. The same machine-learning based segmentation algorithm that the Pixel camera uses in Portrait Mode is used to identify human subjects and add a warm glow to them.

If you’ve used the front facing camera on the Pixel 2 for Portrait Mode selfies, you’ve probably noticed how well it detects and masks human subjects using only segmentation. By using that same segmentation method for synthetic fill flash, the Pixel 3 is able to relight human subjects very effectively, with believable results that don’t confuse and relight other objects in the frame.

Interestingly, the same segmentation methods used to identify human subjects are also used for front-facing video image stabilization, which is great news for vloggers. If you’re vlogging, you typically want yourself, not the background, to be stabilized. That’s impossible with typical gyro-based optical image stabilization. The Pixel 3 analyzes each frame of the video feed and uses digital stabilization to steady you in the frame. There’s a small crop penalty to enabling this mode, but it allows for very steady video of the person holding the camera.

4. Learning-based Portrait Mode

The Pixel 2 had one of the best Portrait Modes we’ve tested despite having only one lens. This was due to its clever use of split pixels to sample a stereo pair of images behind the lens, combined with machine-learning based segmentation to understand human vs. non-human objects in the scene (for an in-depth explanation, watch my video here). Furthermore, dual pixel AF meant robust performance of even moving subjects in low light – great for constantly moving toddlers. The Pixel 3 brings some significant improvements despite lacking a second lens.

According to computational lead Marc Levoy, “Where we used to compute stereo from the dual pixels, we now use a learning-based pipeline. It still utilizes the dual pixels, but it’s not a conventional algorithm, it’s learning based”. What this means is improved results: more uniformly defocused backgrounds and fewer depth map errors. Have a look at the improved results with complex objects, where many approaches are unable to reliably blur backgrounds ‘seen through’ holes in foreground objects:


Learned result. Background objects, especially those seen through the toy, are consistently blurred. Objects around the peripheries of the image are also more consistently blurred.	Learned depth map. Note how objects in the background (blue) aren’t confused as being closer to the foreground (yellow) as they are in the heat map below.

Stereo-only result. Background objects, especially those seen through the toy, aren’t consistently blurred.	Stereo-only based depth map from dual pixels. Note how some elements in the background appear to be closer to the foreground than they really are.

Interestingly, this learning-based approach also yields better results with mid-distance shots where a person is further away. Typically, the further away your subject is, the less difference in stereo disparity between your subject and background, making accurate depth maps difficult to compute given the small 1mm baseline of the split pixels. Take a look at the Portrait Mode comparison below, with the new algorithm on the left vs. the old on the right.


Learned result. The background is uniformly defocused, and the ground shows a smooth, gradual blur.	Stereo-only result. Note the sharp railing in the background, and the harsh transition from in-focus to out-of-focus in the ground.

5. Night Sight

Rather than simply rely on long exposures for low light photography, ‘Night Sight’ utilizes HDR+ burst mode photography to take usable photos in very dark situations. Previously, the Pixel 2 would never drop below 1/15s shutter speed, simply because it needed faster shutter speeds to maintain that 9-frame buffer with zero shutter lag. That does mean that even the Pixel 2 could, in very low light, effectively sample 0.6 seconds (9 x 1/15s), but sometimes that’s not even enough to get a usable photo in extremely dark situations.

The camera will merge up to 15 frames… to get you an image equivalent to a 5 second exposure

The Pixel 3 now has a ‘Night Sight’ mode which sacrifices the zero shutter lag and expects you to hold the camera steady after you’ve pressed the shutter button. When you do so, the camera will merge up to 15 frames, each with shutter speeds as low as, say, 1/3s, to get you an image equivalent to a 5 second exposure. But without the motion blur that would inevitably result from such a long exposure.

Put simply: even though there might be subject or handheld movement over the entire 5s span of the 15 frame burst, many of the the 1/3s ‘snapshots’ of that burst are likely to still be sharp, albeit possibly displaced relative to one another. The tile-based alignment of Google’s ‘robust merge’ technology, however, can handle inter-frame movement by aligning objects that have moved and discarding tiles of any frame that have too much motion blur.

Have a look at the results below, which also shows you the benefit of the wider-angle, second front-facing ‘groupie’ camera:


Normal front-camera ‘selfie’	Night Sight ‘groupie’ with wide-angle front-facing lens

Furthermore, Night Sight mode takes a machine-learning based approach to auto white balance. It’s often very difficult to determine the dominant light source in such dark environments, so Google has opted to use learning-based AWB to yield natural looking images.

Final thoughts: simpler photography

The philosophy behind the Pixel camera – and for that matter the philosophy behind many smartphone cameras today – is one-button photography. A seamless experience without the need to activate various modes or features.

This is possible thanks to the computational approaches these devices embrace. The Pixel camera and software are designed to give you pleasing results without requiring you to think much about camera settings. Synthetic fill flash activates automatically with backlit human subjects, and Super Resolution automatically kicks in as you zoom.

At their best, these technologies allows you to focus on the moment

Motion photos turns on automatically when the camera detects interesting activity, and Top Shot now uses AI to automatically suggest the best photo of the bunch, even if it’s a moment that occurred before you pressed the shutter button. Autofocus typically focuses on human subjects very reliably, but when you need to specify your subject, just tap on it and ‘Motion Autofocus’ will continue to track and focus on it very reliably. Perfect for your toddler or pet.

At their best, these technologies allow you to focus on the moment, perhaps even enjoy it, and sometimes even help you to capture memories you might have otherwise missed.

We’ll be putting the Pixel 3 through its paces soon, so stay tuned. In the meantime, let us know in the comments below what your favorite features are, and what you’d like to see tested.

¹In good light, these last 9 frames typically span the last 150ms before you pressed the shutter button. In very low light, it can span up to the last 0.6s.

²We were only told ‘say, maybe 15 images’ in conversation about the number of images in the buffer for Super Res Zoom and Night Sight. It may be more, it could be less, but we were at least told that it is more than 9 frames. One thing to keep in mind is that even if you have a 15-frame buffer, not all frames are guaranteed to be usable. For example, if in Night Sight one or more of these frames have too much subject motion blur, they’re discarded.

³You can achieve a similar super-resolution effect manually with traditional cameras, and we describe the process here.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

Pixel 3 and Pixel 3 XL feature enhanced computational features, dual front-facing cameras

10 Oct

Google has announced the Pixel 3 and Pixel 3 XL almost exactly a year after their predecessors debuted. The 5.5″ Pixel 3 and 6.3″ XL feature larger displays than the previous generation (5″ and 6″ respectively) but keep roughly the same size and weight.

In a world of dual and triple-camera arrays, Google is staying the course with a single rear-facing camera on each device: the same 12.2MP sensor with dual pixel autofocus and 28mm equiv. F1.8 aperture that appeared in the Pixel 2 and 2 XL. However, two front-facing cameras are now offered: a 19mm equiv. 8MP F2.2 with fixed focus designed for group selfies, and a 28mm equiv. 8MP F1.8 with phase detection autofocus.

$ (document).ready(function() { SampleGalleryV2({“containerId”:”embeddedSampleGallery_0806398624″,”galleryId”:”0806398624″,”isEmbeddedWidget”:true,”selectedImageIndex”:0,”isMobile”:false}) });

The real story, as it tends to be lately, is the software. Working with just one main camera, Google has doubled down on computational solutions to physical limitations. But there’s a silver lining to this approach: any techniques that make a single camera better will eventually make multiple cameras that much better once that approach is (arguably inevitably) adopted.

The Pixel 2 shot and aligned up to 9 frames for every image taken to maximize detail and reduce noise, and the Pixel 3 is now capable of shooting, buffering, and aligning up to 15 frames per shot. All still with zero shutter lag – you get the shot that represents the instant you hit the shutter button.

A new ‘Night Sight’ feature that combines multiple frames with long shutter speeds for extremely low light shots

There’s a catch, for now though. These extra frames are only used when zooming your image 1.2x or more, or when the environment is so dark as to require longer shutter speeds. That latter feature is called ‘Night Sight’ and it combines multiple frames with long shutter speeds for extremely low light shots. It does this using Google’s ‘robust merge’, which is able to effectively deal with subject movement without blur or ghosting.

Another computational feature called ‘synthetic fill flash’ understands human subjects and raises their exposure with a fill-flash effect. The result is often a nice warm glow on faces, particularly in backlit situations where they might otherwise be rendered dark.

Google uses super-resolution techniques to tackle the problem of poor image quality with digital zoom. By capturing multiple frames with sub-pixel resolution, the Pixel 3 can record detail finer than traditional approaches, which means that digitally zoomed shots, which crop and enlarge smaller portions of the frame, can – we’re told – compete with optical zoom approaches.

Particularly innovative about this approach is the removal of the need to demosaic: with pixel-level image alignment the Pixel 3 can combine images that have been off-set by one pixel shifts, which means that every color has been sampled at each pixel position in the final frame. No demosaicing means sharper images with less noise.

Official Google Pixel 3 sample images

$ (document).ready(function() { SampleGalleryV2({“containerId”:”embeddedSampleGallery_5730154109″,”galleryId”:”5730154109″,”isEmbeddedWidget”:true,”selectedImageIndex”:0,”isMobile”:false}) });

A new Top Shot feature is available when taking motion photos – the device constantly buffers images and captures alternates, using AI to suggest the best photo of the bunch, even if it was captured before the shutter was pressed.

Improvements have also been made to Portrait Mode. Google says its depth mapping is better, with a new learning-based approach that is better at judging background and foreground objects. The result is fewer depth map errors, more uniform blur across the frame, and more natural transitions from in focus to out-of-focus areas. The level of blur and point of focus can be changed after the fact. Continuous subject tracking is now available as well – tap a subject and the camera will track and maintain focus on it, in stills or video.

Google says its depth mapping is better, with a new learning-based approach that is better at judging background and foreground objects

In non-photographic improvements, the Pixel 3 and 3 XL boast more robust waterproofing with an IP68 rating. And this year neither display appears to have the viewing angle and hue shift issues of last year’s Pixel 2 XL. In our use so far, the displays appear to be right up there with the best we’ve seen.

Both the Pixel 3 and Pixel 3 XL will be offered in Just Black, Clearly White and Not Pink color variations, in either 64 or 128GB. The Pixel 3 starts at $ 799 and the 3 XL starts at $ 899.

Google Pixel 3. Make every day more extraordinary.

Today we’re introducing Pixel 3 and Pixel 3XL, the new smartphones from Google. Pixel brings you the best of Google in a phone, powered by AI to deliver more helpful, thoughtful, and enjoyable experiences. That means a phone that answers for you when a telemarketer calls, a camera that uses AI to make sure you never miss the shot, and a more helpful visual and audio experience while charging, powered by the Google Assistant.

Brilliant photos every time and super-charged selfies
We’re taking more photos on our phones than ever before, but we still often miss the perfect moment. Pixel 3 helps you get that perfect shot on the first try.

$ (document).ready(function() { SampleGalleryV2({“containerId”:”embeddedSampleGallery_5730154109_1″,”galleryId”:”5730154109″,”isEmbeddedWidget”:true,”selectedImageIndex”:0,”isMobile”:false}) });

Here’s how the best camera gets even better with Pixel 3:

Capture smiles, not blinks: A feature we call Top Shot uses AI to help you capture the perfect photo every time. When you take a motion photo, it captures alternate shots in HDR+, then recommends the best one—even if it’s not exactly when you hit the shutter, looking for those where everyone is smiling, with eyes open, and facing the camera. Top Shot automatically captures alternate shots in HDR+. If your timing wasn’t perfect, the camera will suggest a better one and give you the option to save it.
Get better zoom: When you zoom in on a phone camera, the image looks grainy. Super Res Zoom is a computational photography technique, traditionally used for astronomy and scientific imaging, that produces sharp details when you zoom.
No light; no problem: Pixel 3 lets you take natural-looking photos in dark surroundings, all without a flash. With Night Sight, coming soon to Pixel 3, you can take bright, detailed, colorful shots around the campfire, in a moonlit forest, or a selfie after you close out the bar.
No selfie stick required: Get everyone in the picture with Group Selfie, which gives you 184% more room in your photo for friends and scenery.
Look … no hands! Photobooth mode uses AI to recognize that when you’re smiling or making a funny expression, you’re ready for a selfie. It snaps the photo on its own so that you don’t need to reach for the shutter button—a good option for candids.
Even more stunning portraits, front and back: When you take photos in Portrait Mode, you can change the blurriness of the background, or change the part of the picture in focus, after the fact. Google Photos can also make the subject of your photo pop by leaving them in color, while changing the background to black and white.
Create and play: In Playground, you can make photos, selfies, and videos come to life by adding your favorite superheroes, animated stickers, and fun captions. In celebration of Marvel Studios’ 10 Year Anniversary, you’ll enjoy seeing the characters from the Marvel Cinematic Universe (exclusively on Pixel) react to each other and to you.
Super smooth video: When you want to capture something that won’t stop moving—think an adorable toddler or your new puppy—Motion Auto Focus will make sure your Pixel 3 camera stays in sharp focus automatically, as you record. And if you happen to be taking a selfie video while walking or moving around, Pixel 3 brings you front-facing video stabilization.

Unlimited storage for all of your photos and videos
With Pixel 3, you can save all your favorite moments with free, unlimited photo and video storage in original resolution. It’s hassle-free, you don’t have to think about back-ups. Come back to Google Photos later and search for the beach photos you took on your Pixel 3, and they’ll pop right up.

Your AI-powered sidekick
The AI in Pixel 3 enables new features that make your day-to-day actions simpler and easier.

If you want to know more about something you’re looking at, use Google Lens, built right into the Pixel 3 camera. To scan and translate text, find similar styles of clothing, or identify popular plants and animals, you can now long press in the Pixel 3 camera to easily open Lens. When you point your camera at information you want to remember or don’t feel like typing in—like a URL or QR code on a flyer or an email address on a business card—Google Lens suggests what to do next, like creating a new contact.

You can count on even more help across other apps too, including Gmail’s Smart Compose, now available for mobile on Pixel 3. Smart Compose suggests phrases in your emails so that you can draft them faster, on the go. Gboard, the keyboard built into your Pixel 3, will recommend GIFs, stickers, and more, to make your conversations fun and engaging. Available first in English.

The Google Assistant is also baked into Pixel 3 to help you find answers and control your phone and compatible smart home devices—all with a simple squeeze or just by using your voice. This year we have two new Assistant features coming to Pixel:

First, Pixel 3’s on-device AI helps you screen phone calls and avoid spam calls. Imagine you’re at dinner with family or in a meeting at work and a call from an unknown caller comes in. Just tap on “Screen call” to find out who’s calling and why, as well as other information (as prompted by you). You’ll immediately see a transcript of the caller’s responses so that you can then decide whether to pick up, respond by tapping a quick reply (e.g. “I’ll call you back later”), or mark the call as spam and dismiss. Processing the call details on-device means these experiences are fast, private to you, and use up less battery.

Second, Pixel users also get help with making calls. Later this year, Pixel users will be the first to get access to an experimental new Google Assistant feature, powered by Duplex technology, which helps you complete real-world tasks over the phone, like calling a restaurant to book a table. This feature will initially be available in New York, Atlanta, Phoenix and San Francisco to help people book restaurant reservations and will roll out to other cities in the future.

As we develop new calling technologies, we believe it’s critical that we help users understand the context of the conversation. We’ll disclose to businesses receiving the call that they’re speaking to an automated system, and we have developed controls to protect against spam and abuse, as well as the ability for a business to opt-out of receiving calls. For Call Screen, we will also let the caller know that a screening service is being used.

Digital Wellbeing
Our phones, while probably the most important tech in our lives, shouldn’t control our lives. So Digital Wellbeing, a suite of tools to help you find your own balance with technology, is built into Pixel 3. It includes a dashboard to help you understand how you spend time on your phone, the ability to set time limits on specific apps, and a new Wind Down mode to help you get to sleep at night by gently transitioning your display to a grayscale screen. When you don’t want to be bothered by rings or notifications, just flip to Shhh— an easy gesture that turns on Do Not Disturb and minimizes distractions.

Fast and wireless charging
Pixel 3 comes with an 18 Watt fast charger in the box, which can give you 7 hours of use in 15 minutes of charging. With our AI-powered Adaptive Battery technique, Pixel 3 prioritizes battery power for your most important apps to make your phone last all day.

Alongside Pixel 3, we’re also introducing Pixel Stand, our new, Qi compliant wireless charger (sold separately). While charging in the Pixel Stand, your phone turns into a smart visual and audio experience powered by the Google Assistant, similar to Google Home Hub. It answers your questions, plays music, helps you control smart home devices, transitions into a photo frame when idle, and much more. If you set an alarm, your screen will gently brighten over 15 minutes before your alarm goes off, mimicking the sunrise and helping you wake up naturally.

Pixel 3 is IP68 water and dust resistant and has a security chip custom-designed by Google called Titan M, making it the most secure phone we’ve built yet. Titan M enhances mobile security by protecting your unlock credentials, disk encryption, app data, and the integrity of the operating system code itself. Powered by Android 9 Pie, Pixel 3 comes with the latest Android operating system.

You can choose from two sizes – the 5.5” Pixel 3 and the 6.3” Pixel 3 XL – and three colors – Just Black, Clearly White, and Not Pink. Both have the exact same feature set and include a high quality Pixel USB-C earbuds and a USB-C Digital to 3.5 mm headphone adapter in the box. Pixel 3 comes with dual front-firing speakers tuned by a GRAMMY award-winning music producer to turn your phone into a powerful speaker. Customers who activate a Pixel 3 or Pixel 3 XL by December 31, 2018 can get 6-months of free YouTube Music Premium.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

The iPhone XS is a leap forward in computational photography

05 Oct

Aside from folks who still shoot film, almost nobody uses the term ‘digital photography’ anymore – it’s simply ‘photography,’ just as we don’t keep our food in an ‘electric refrigerator.’ Given the changes in the camera system in Apple’s latest iPhone models, we’re headed down a path where the term ‘computational photography’ will also just be referred to as ‘photography,’ at least by the majority of photographers.

The iPhone XS and iPhone XS Max feature the same dual-camera and processing hardware; the upcoming iPhone XR also sports the same processing power, but with only a single camera: the same wide-angle F1.8 one on the other models. The image sensor captures 12 megapixels of data, the same resolution as every previous model dating back to the iPhone 6s, but the pixels themselves are larger at 1.4 µm, compared to 1.22 µm for the iPhone X, meaning a slightly larger sensor. (For more on the camera’s specs, see “iPhone XS, XS Max, and XR cameras: what you need to know.”)

$ (document).ready(function() { SampleGalleryV2({“containerId”:”embeddedSampleGallery_7780769194″,”galleryId”:”7780769194″,”isEmbeddedWidget”:true,”selectedImageIndex”:0,”isMobile”:false}) });

More important this year is upgraded computational power and the software it enables: the A12 Bionic processor, the eight-core ‘Neural Engine,’ and the image signal processor (ISP) dedicated to the camera functions. The results include a new Smart HDR feature that rapidly combines multiple exposures for every capture, and improved depth-of-field simulation using Portrait mode. (All the examples throughout are straight out of the device.)

Smart HDR

This feature intrigued me the most, because last year’s iPhone 8, iPhone 8 Plus and iPhone X introduced HDR as an always-on feature. (See “HDR is enabled by default on the iPhone 8 Plus, and that’s a really good thing.”) HDR typically blends two or more images of varying exposures to end up with a shot with increased dynamic range, but doing so introduces time as a factor; if objects are in motion, the delay between captures makes those objects blurry. Smart HDR captures many interframes to gather additional highlight information, and may help avoid motion blur when all the slices are merged into the final product.

The iPhone XS image almost looks as if it was shot using an off-camera flash

Testing Smart HDR proved to be a challenge, because unlike with the HDR feature in earlier models, the Photos app doesn’t label Smart HDR images as such. After shooting in conditions that would be ripe for HDR – bright backgrounds and dark foreground, low-light conditions at dusk – nothing had that HDR indicator. I wasn’t initially sure if perhaps the image quality was due to Smart HDR or the larger sensor pixels; no doubt some credit is due to the latter, but it couldn’t be that much.

Comparing shots with those taken with an iPhone X reveals Smart HDR at work, though. In the following photo at dusk, I wanted to see how well the cameras performed in the fading light and also with motion in the scene (the flying sand). The iPhone X image is dark, but you still get a fair bit of detail in the girl’s face and legs, which are away from the sun. The iPhone XS image almost looks as if it was shot using an off-camera flash, likely because the interframes allow highlight retention and motion freezing even as ‘shutter speeds’ become longer.

Shot with iPhone X

Shot with iPhone XS

As another example, you can see the Smart HDR on the iPhone XS working in even darker light compared to the iPhone X shot. At this point there’s more noise in both images, but it’s far more pronounced in the iPhone X photo.


Shot with iPhone X	Shot with iPhone XS

Smart HDR doesn’t seem to kick in when shooting in burst mode, or the effect isn’t as pronounced. Considering the following photo is captured at 1/1000 sec, and the foreground isn’t a silhouette, the result isn’t bad.

iPhone XS image shot in burst mode. It’s dark, but picks up the detail in the sand.

iPhone XS image shot in burst mode.

iPhone XS non-burst image captured less than a minute after the photo above.

Portrait Mode

The iPhone’s Portrait mode is a clever cheat involving a lot of processing power. On the iPhone X and iPhone 8 Plus, Apple used the dual backside cameras to create a depth map to isolate a foreground subject – usually a person, but not limited to people-shaped objects – and then blur the background based on depth. It was a hit-or-miss feature that sometimes created a nice shallow depth-of-field effect, and sometimes resulted in laughable, blurry misfires.

On the iPhone XS and iPhone XS Max, Apple augments the dual cameras with Neural Engine processing to generate better depth maps, including a segmentation mask that improves detail around the edge of the subject. It’s still not perfect, and one pro photographer I know immediately called out what he thought was a terrible appearance, but it is improved, and in some cases most people may not recognize that it’s all done in software.

The notable addition to Portrait mode in the iPhone XS and iPhone XS Max is the ability to edit the simulated depth of field within the Photos app. A depth control slider appears for Portrait mode photos, with f-stop values from F1.4 to F16. The algorithm that creates the blur also seems improved, creating a more natural effect than a simple Gaussian blur.

Apple also says it’s analyzed the optical characteristics of some “high-end lenses” and tried to mimic their bokeh. For instance, the simulated blue should produce circular discs at the center of the image but develop a ‘cats-eye’ look as you approach the edge of the image. The company says that a future update will include that control in the Camera app for real-time preview of the effect.

Portrait mode is still no substitute for optics and good glass. Sometimes objects appear in the foreground mask – note the coffee cup over the shoulder at left in the following image – and occasionally the processor just gets confused, blurring the horizontal lines of the girl’s shirt in the next example. But overall, you can see progress being made toward better computational results.

Flare and a Raw Footnote

One thing I noticed with my iPhone XS is that it produced more noticeable lens flare when catching direct light from the sun or bright sources such as playing-field lights, as in the following examples; notice the blue dot pattern in the foreground of the night image.

Since I wanted to focus on the Smart HDR and Portrait mode features for this look, I haven’t shot many Raw photos using third-party apps such as Halide or Manual (the built-in Photos app does not include a Raw capture mode). Sebastiaan de With, the developer of Halide, determined that in order to make faster captures, the camera is shooting at higher ISOs, and then de-noising the results via software. With Raw photos, however, that results in originals that aren’t as good as those created by the iPhone X, because they’re noisier and exposed brighter. You can read more at the Halide blog: iPhone XS: Why It’s a Whole New Camera.

Overall, though, the camera system in the iPhone XS and iPhone XS Max turn out to be larger improvements than they initially seemed, especially for the majority of iPhone owners who want to take good photos without fuss. Apple’s computational photography advancements in these models deliver great results most of the time, and point toward more improvements in the future.

iPhone XS sample gallery

$ (document).ready(function() { SampleGalleryV2({“containerId”:”embeddedSampleGallery_7780769194_1″,”galleryId”:”7780769194″,”isEmbeddedWidget”:true,”selectedImageIndex”:0,”isMobile”:false}) });
Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized