360Photography

Lytro debuted its Cinema prototype to an eager crowd at NAB 2016 in Las Vegas, NV.

Lytro greeted a packed showroom at NAB 2016 in Las Vegas, Nevada to demo its prototype Lytro Cinema camera and platform, as well as debut footage shot on the system. To say we’re impressed from what we saw would be an understatement: Lytro may be poised to change the face of cinema forever.

The short film ‘Life’, containing footage shot both on Lytro Cinema as well as an Arri Alexa, demonstrated some of the exciting applications of light field in video. Directed by Academy Award winner Robert Stromberg and shot by VRC Chief Imaging Scientist David Stump, ‘Life’ showcased the ability of light field to obviate green screens, allowing for extraction of backgrounds or other scene elements based off of depth information, and seamless integration of CGI elements into scenes. Lytro calls it ‘depth screening’, and the effect looked realistic to us.

Just as exciting was the demonstration of a movable virtual camera in post: since the light field contains multiple perspectives, a movie-maker can add in camera movement at the editing stage, despite using a static camera to shoot. And we’re not talking about a simple pan left/right, up/down, or a simple Ken Burns effect… we’re talking about actual perspective shifts. Up, down, left, right, back and forth, even short dolly movements – all simulated by moving a virtual camera in post, not by actually having to move the camera on set. To see the effect, have a look at our interview with Ariel Braunstein of Lytro, where he presents a camera fly-through from a single Lytro Illum shot (3:39 – 4:05):

The Lytro Cinema is capable of capturing these multiple perspectives because of ‘sub-aperture imaging’. Head of Light Field Video Jon Karafin explains that the system is made of multiple lenses (we see what appears to be two separate openings in the photo below), and behind each lens, in front of the sensor, is a microlens array consisting of millions of small lenses similar to what traditional cameras have. The difference, though, is that there is a 6×6 pixel array underneath each microlens, meaning that any one XY position of those 36 pixels under one microlens, when combined with the same position pixel under all other microlenses, represents the scene as seen through one portion, or ‘sub-aperture’ of the lens. These 36 sub-aperture images essentially provide 36 different perspectives, which then allow for computational reconstruction of the image with all the benefits of light field.

The 36 different perspectives affords you some freedom of movement in moving a virtual camera in post, but it is of course limited, affected by considerations like lens, focal length, and subject distance. It’s not clear yet what that range of freedom is with the Cinema, but what we saw in the short film was impressive, something cinematographers will undoubtedly welcome in place of setting up motion rigs for small camera movements. Even from a consumer perspective, consider what auto-curation of user-generated content could do with tools like these. Think Animoto on steroids.

Front of the Lytro Cinema, on display at NAB 2016. We see two openings, though it’s not clear how many main imaging lenses exist in the prototype yet.

We’ve focused on depth screening and perspective shift, but let’s not forget all the other benefits light field brings. The multiple perspectives captured mean you can generate 3D images or video from every shot at any desired parallax disparity (3D filmmakers often have to choose their disparity on-set, only able to optimize for one set of viewing conditions). You can focus your image after the fact, which saves critical focus and focus approach (its cadence) for post.* Selective depth-of-field is also available in post: choose whether you want shallow, or extended, depth-of-field, or even transition from selective to extensive depth-of-field in your timeline. You can even isolate shallow or extended depth-of-field to different objects in the scene using focus spread: say F5.6 for a face to get it all in focus, but F0.3 for the rest of the scene.

Speaking of F0.3 (yes, you read that right), light field allows you to simulate faster (and smaller) apertures previous thought impossible in post, which in turn places fewer demands on lens design. That’s what allowed the Illum camera to house a 30-250mm equiv. F2.0 constant aperture lens in relatively small and lightweight body. You could open that aperture up to F1.0 in post, and at the demo of Cinema at NAB, Lytro impressed its audience with – we kid you not – F0.3 depth-of-field footage.

But all this doesn’t come without a cost: the Lytro Cinema appears massive, and rightfully so. A 6×6 pixel array underneath each microlens means there are 36 pixels for every 1 pixel on a traditional camera; so to maintain spatial resolution, you need to grow your sensor, and your total number of pixels. Which is exactly what Lytro did – the sensor housing appeared to our eyes to be over a foot in width, sporting a whopping 755 million total pixels. The optics aren’t small either. The total unit lives on rails on wheels, so forget hand-held footage – for now. Bear in mind though, the original technicolor cinematic camera invented back in 1932 appeared similarly gargantuan, and Lytro specifically mentioned that different versions of Cinema are planned, some smaller in size.

The Lytro Cinema is massive. The sensor is housed in the black box behind the orange strut, which appears to be at least a foot wide. It comes with its own traveling server to deal with the 300GB/s data rates. Processing takes place in the cloud where Google spools up thousands of CPUs to compute each thing you do, while you work with real-time proxies.

So what does 755MP get you? A lot of data, for starters. We spoke to Lytro some time back about this, and were told that the massive sensor requires a bandwidth of around 300GB/s. That means Lytro Cinema comes with its own server on-set to capture that data. But processing that data isn’t easy either – in fact, no mortal laptop or desktop need apply. Lytro is partnering with Google to send footage to the cloud, where thousands of CPUs crunch the data and provide you real-time proxies for editing. One major concern with Lytro’s previous cameras was the resolution trade-off: recording angular information means that spatial resolution is sacrificed. The Illum had a roughly 40MP sensor, yet yielded only roughly 5MP images, a roughly 10-fold resolution cost. With 755MP though, even a 10x resolution cost would yield 76MP – well above the requirements for 4K video.**

Speculation aside, the 4K footage from the Lytro Cinema that was mixed with Arri Alexa footage to create the short ‘Life’, viewed from our seating position, appeared comparable to what one might expect from professional cinema capture. Importantly, the footage appeared virtually noise free – which one might expect of such a large sensor area. Since image data from many pixels are used for any final image pixel, a significant amount of noise averaging occurs – yielding a clean image, and a claimed 16 stops of dynamic range.

That’s incredibly impressive, given all the advantages light field brings. This may be the start of something incredibly transformative for the industry. After all, who wouldn’t want the option for F0.3 depth-of-field with perfect focus in post, adjustable shutter angle, compellingly real 3D imagery when paired with a light field display, and more? With increased capabilities for handling large data bandwidths, larger sensors, and more pixels, we think some form of light field will exist perhaps in most cameras of the future. Particularly when it comes to virtual reality capture, which Lytro also intends to disrupt with Immerge.

It’s impressive to witness how far Lytro has come in such a short while, and we can’t wait to see what’s next. For more information, visit Lytro Cinema.

* If it’s anything like the Illum, though, some level of focusing will still be required on set, as there are optimal planes of refocus-ability.

** We don’t know what the actual trade-off is for the current Lytro Cinema. It’s correlated to the number of pixels underneath each microlens, and effective resolution can change at different focal planes.

Articles: Digital Photography Review (dpreview.com)