Research « 360Photography

RSS

Posts Tagged ‘Research’

NVIDIA Research develops a neural network to replace traditional video compression

06 Oct

NVIDIA researchers have demonstrated a new type of video compression technology that replaces the traditional video codec with a neural network to drastically reduce video bandwidth. The technology is presented as a potential solution for streaming video in situations where Internet availability is limited, such as using a webcam to chat with clients while on a slow Internet connection.

The new technology is made possible using NVIDIA Maxine, a cloud-AI video streaming platform for developers. According to the researchers, using AI-based video compression can strip video bandwidth usage down to 1/10th of the bandwidth that would otherwise be used by the common H.264 video codec. For users, this could result in what NVIDIA calls a ‘smoother’ experience that uses up less mobile data.

In a video explaining the technology, researchers demonstrate their AI-based video compression alongside H.264 compression with both videos limited to the same low bandwidth. With the traditional video compression, the resulting low-bandwidth video is very pixelated and blocky, but the AI-compressed video is smooth and relatively clear.

This is made possible by extracting the key facial points on the subject’s face, such as the position of the eyes and mouth, then sending that data to the recipient. The AI technology then reconstructs the subject’s face and animates it in real time using the keypoint data, the end result being very low bandwidth usage compared to the image quality on the receiver’s end.

There are some other advantages to using AI-based compression that exceed the capabilities of traditional video technologies, as well. One example is Free View, a feature in which the AI platform can rotate the subject so that they appear to be facing the recipient even when, in reality, their camera is positioned off to the side and they appear to be staring into the distance.

Likewise, the keypoints extracted from the subject’s face could also be used to apply their movements to other characters, including fully animated characters, expanding beyond the AI-powered filters that have become popular some video apps like Snapchat. Similar technology is already on the market in the form of Apple’s AI-based Animoji.

The use of artificial intelligence to modify videos isn’t new; most major video conferencing apps now include the option of replacing one’s real-life background with a different one, including intelligent AI-based background blurring. However, NVIDIA’s real-time AI-based video compression takes things to a new level by using AI to not only generate the subject in real time, but also modify them in convenient ways, such as aligning their face with a virtual front-facing camera.

The technology could usher in an era of clearer, more consistent video conferencing experiences, particularly for those on slow Internet connections, while using less data than current options. However, the demonstration has also raised concerns that largely mirror ones related to deepfake technologies — namely, the potential for exploiting such technologies to produce inauthentic content.

Artificial intelligence technology is advancing at a clipped rate and, in many cases, can be used to imperceptibly alter videos and images. Work is already underway to exceed those capabilities, however, by fully generating photo-realistic content using AI rather than modifying existing real-world content.

The Allen Institute for AI recently demonstrated the latest evolution in this effort by using both images and text to create a machine learning algorithm that possesses a very basic sense of abstract reasoning, for example. NVIDIA Research has also contributed extensively to this rapidly evolving technology, with past demonstrations including generating landscapes from sketches, generating photo-realistic portraits and even swapping facial expressions between animals.

A number of companies are working to develop counter technologies capable of detecting manipulated content by looking for markers otherwise invisible to the human eye. In 2019, Adobe Research teamed up with UC Berkeley to develop and demonstrate an AI capable of not only identifying portrait manipulations, but also automatically reversing the changes to display the original, unmodified content.

The general public doesn’t yet have access to these types of technologies, however, generally leaving them vulnerable to the manipulated media that permeates social media.

Via: NVIDIA

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

Disney Research Studios demonstrates automatic face swapping with faster, cheaper AI

21 Jul

Disney Research Studios and ETH Zurich have published a study detailing a new algorithm that is able to swap faces from one subject to another in high-resolution photos and videos. Of note, this system is able to fully automate the face-swapping process, presenting the first instance of megapixel-resolution machine-generated imagery that is ‘temporally coherent’ and photo-realistic.

The new algorithm involves taking the face of a subject and modifying it using the face of another person, blending the two so that the face from one person is presented with the expressions and movements of another.

The system involves a multi-way comb network trained with images of multiple people, as well as a blending method that preserves contrast and lighting. ‘We also show that while progressive training enables generation of high-resolution images,’ the researchers say, ‘extending the architecture and training data beyond two people allows us to achieve higher fidelity in generated expressions.’

Key to the high level of quality is the ‘landmark stabilization algorithm,’ which Disney researchers describe as a ‘crucial’ aspect of dealing with high-resolution content. Though this isn’t the first instance of face-swapping in footage, the study points out that existing methods used to generate characters like the young Carrie Fisher in Rogue One are both time-intensive and quite expensive.

Artificial intelligence has the potential to change this, ultimately enabling creators to rapidly generate computer characters using live-action footage and input images of the target. Generating realistic faces remains a big problem, however, producing what is referred to as the ‘uncanny valley’ look that limits the use of this tech.

This makes Disney’s new technology particularly exciting, teasing a future in which creators will be able to generate photo-realistic, high-resolution, temporally-stable face swaps between two people. The researchers explain:

As our system is also capable of multi-way swaps — allowing any pair of performances and appearances in our data to be swapped — the possible benefits to visual effects are extensive, all at a fraction of the time and expense required using more traditional methods.

The study compares the face-swapping results from this new method to the results from existing algorithms, including DeepFaceLab and DeepFakes. Though the other algorithms were able to produce casually convincing results, they were unable to pass scrutiny and, in some cases, were either excessively blended or outright bizarre and uncanny.

This batch represents instances of failed face swapping

In comparison, the face swaps generated using the new method were realistic and maintained a high level of sharpness and detail at a 1024 x 1024 resolution, bypassing the soft, blurry results often seen when using DeepFakes. As well, the researchers note that DeepFakes has such heavy processing requirements that it was only able to generate a resolution of 128 x 128 pixels using an 11GB GPU.

When using morphable models, the researchers were able to increase the resolution to 500 x 500 pixels, but the results were typically unrealistic. Beyond that, the researchers were forced to train the conventional models for each pair of face swaps whereas the new algorithm could be simultaneously trained for all of the people used for the various face swaps.

However, the study points out that the new algorithm presents one big limitation also experienced by other, more conventional methods: the original head shape is maintained. Though the face swap may be very realistic, the face itself may not match the head shape properly, resulting in a generated character that looks a bit ‘off’ from what is expected.

Future research may result in a method for transferring the subject’s head shape in addition to their face, producing not only photo-realistic results, but also the correct overall appearance for a digitally-recreated actor. The biggest obvious use for this technology is in film and television, enabling studios to quickly and cheaply (relatively speaking) create 3D models of aging or deceased actors.

This technology joins a growing body of research on face-swapping and model-generating algorithms that focus on still images rather than videos. NVIDIA, for example, published a study in late 2018 that demonstrated the generation of photo-realistic portraits of AI models that involved source and target images of real people.

Around a year later, the same company published new research that performed a similar face swap, but one involving dogs instead of humans. We’ve already seen the use of these various AI technologies reach the consumer level — Let’s Enhance 2.0, for example, recently introduced a new feature that utilizes machine learning to reconstruct the faces of subjects in low-resolution images.

As for the new study from Disney Research Studios and ETH Zurich, the full paper (PDF) can be found on Disney’s website here.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

Research firm claims Sony had nearly half of the image sensor market share in 2019

18 Feb

Sony held nearly half of the image sensor market share in 2019, according to Japanese research firm Techno Systems Research (TSR). The detail was spied by PulseNews, which points out that Sony’s 49.1% market share greatly eclipsed the second biggest market share, 17.9%, held by Samsung. The South Korean company recently launched its 108MP Nonacell image sensor with its new Galaxy S20 Ultra smartphone.

The news isn’t surprising. In December, Sony revealed that its semiconductor business was working 24/7 through the holidays in an effort to keep up with the demand for its image sensors. The company is building a new facility in Nagasaki in order to boost production capacity due to this demand; it is expected to go online in April 2021.

Samsung remains the biggest competitor to Sony’s image sensor business. In 2018, the South Korean company announced that it would expand the production capacity of its own image sensor business and that its ultimate goal was to overtake Sony. Based on the TSR data, the company still has a long way to go toward reaching that milestone.

Still, Sony’s image sensor business may have a rocky future. Earlier this month, Sony expressed concerns about the ongoing novel coronavirus outbreak in China where the company has four large factories. According to Sony CFO Hiroko Totoki via the Nikkei Asian Review, the impact of the coronavirus on the company’s supply chains could potentially cause enormous disruption to Sony’s image sensor business.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

Vision Research launches its latest high-speed camera, the Phantom VEO 1310

10 Feb

Vision Research has released its latest high-speed camera, the Phantom VEO 1310. The new camera, which is part of Vision Research’s robust ‘VEO’ lineup, can capture 720p video at up to 14,350 frames per second (fps).

The new camera isn’t the highest-resolution phantom on the market, but it still offers plenty in the framerate department. Below is a list of the framerates and resolutions the Phantom VEO 1310 can record at:

1280 x 960 at 10,860 fps
1280 x 720 at 14,350 fps
960 x 960 at 13,333 fps
640 x 480 at 30,030 fps
320 x 120 at 423,350 fps

The camera features a native ISO of 25,000 D in Mono and 6,400 D in Color mode. It offers 18 µm pixel size, 12-bit color depth and has a minimum global shutter framerate of 50 fps.

The Phantom VEO 1310 comes in two models: Light (L) and Full (S). Both models offer SDI and HDMI video out, 12V battery input and include the option to add on a 10Gb ethernet adapter for remote operation. The difference between the two models is that the ’S’ version of the VEO 1310 offers six extra I/O ports (F-sync, TC in/out, trigger, strobe and a ready port), includes a CFast 2.0 port and offers on-camera controls, whereas the ‘L’ version lacks all of the above.

On both models, the lens mount is user-changeable with options for C-mount, Canon EF-mount and PL-mount lenses, with full electronic control support. The camera is made in the United States and comes with a handle, cheese plate, battery mounts (with the ’S’ model) and a case with custom foam cutouts.

Details on pricing and availability are unknown at this time. We have contacted Vision Research and will update this article with more information if and when we receive it.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

Adobe Research and UC Berkeley create AI that can find and undo portrait manipulations

15 Jun

Researchers with Adobe Research and UC Berkeley are working together on the development of a method for identifying photo edits made using Photoshop’s Face Aware Liquify tool. The work is sponsored by DARPA’s MediFor program, which funds researchers who are working to ‘level the digital imagery playing field’ by developing tech that assesses the ‘integrity’ of an image.

Both DARPA and Adobe highlight the issue of readily available image manipulation technologies, including some tools that are offered by select Adobe software. The company says that despite being ‘proud of the impact’ these tools have had, it also recognizes ‘the ethical implications of our technology.’

Adobe said in a blog post on Friday:

Trust in what we see is increasingly important in a world where image editing has become ubiquitous – fake content is a serious and increasingly pressing issue. Adobe is firmly committed to finding the most useful and responsible ways to bring new technologies to life – continually exploring using new technologies, such as artificial intelligence (AI), to increase trust and authority in digital media.

As such, Adobe Research and UC Berkeley researchers have published a new study detailing a method for detecting image warping edits that have been applied to images of human faces. The technology involves a Convolutional Neural Network (CNN) trained on manipulated images that were created using scripts with Photoshop and its Face Aware Liquify tool.

To ensure the method can detect the types of manipulations performed by humans, the image dataset used to train the AI also included some images that were altered by a human artist. ‘This element of human creativity broadened the range of alterations and techniques used for the test set beyond those synthetically generated images,’ the study explains.

To test the deep learning method’s assessment skills, the researchers used image pairs featuring the original unedited image and the image that had been altered. Humans presented with these images could only detect which had been altered with 53% accuracy, whereas the neural network was able to pick the manipulated image with accuracy as high as 99%.

In addition, and unlike the average Photoshop user, the technology is able to pinpoint the specific areas of a face that had been warped, which methods of warping had been used, and calculate the best way to revert the image back to as close to its original state as possible.

Adobe researcher Richard Zhang explained, ‘The idea of a magic universal ‘undo’ button to revert image edits is still far from reality. But we live in a world where it’s becoming harder to trust the digital information we consume, and I look forward to further exploring this area of research.’

The research is described as still in its ‘early stages,’ and is only one part of Adobe’s body of work on image integrity and authenticity. The results come amid the growing sophistication of artificial intelligence technologies capable of generating highly realistic portraits and performing complex edits to images.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

NVIDIA Research project uses AI to instantly turn drawings into photorealistic images

21 Mar

NVIDIA Research has demonstrated GauGAN, a deep learning model that converts simple doodles into photorealistic images. The tool crafts images nearly instantaneously, and can intelligently adjust elements within images, such as adding reflections to a body of water when trees or mountains are placed near it.

The new tool is made possible using generative adversarial networks called GANs. With GauGAN, users select image elements like ‘snow’ and ‘sky,’ then draw lines to segment an image into different elements. The AI automatically generates the appropriate image for that element, such as a cloudy sky, grass, and trees.

As NVIDIA reveals in its demonstration video, GauGAN maintains a realistic image by dynamically adjusting parts of the render to match new elements. For example, transforming a grassy field to a snow-covered landscape will result in an automatic sky change, ensuring the two elements are compatible and realistic.

GauGAN was trained using millions of images of real environments. In addition to generating photorealistic landscapes, the tool allows users to apply style filters, including ones that give the appearance of sunset or a particular painting style. According to NVIDIA, the technology could be used to generate images of other environments, including buildings and people.

Talking about GauGAN is NVIDIA VP of applied deep learning research Bryan Catanzaro, who explained:

This technology is not just stitching together pieces of other images, or cutting and pasting textures. It’s actually synthesizing new images, very similar to how an artist would draw something.

NVIDIA envisions a tool based on GauGAN could one day be used by architects and other professionals who need to quickly fill a scene or visualize an environment. Similar technology may one day be offered as a tool in image editing applications, enabling users to add or adjust elements in photos.

The company offers online demos of other AI-based tools on its AI Playground.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

New Google research thwarts automatic watermark removal

19 Aug

Warped watermarks leave visible artifacts when removed automatically. Image: Google

Watermarks are widely used by photographers and stock agencies to protect their digital property in an online world where very little stands between an eager image thief and your photography. However, even complex watermarks might not be as secure as you’d think when the same pattern is applied to a large number of accessible images.

A research team at Google recently embarked on a two-part experiment. First, they developed a method of quickly,. effectively and automatically removing watermarks from a large set of images. Then, they found a way to thwart their own automatic system, creating a more secure way to watermark.

Automatic Watermark Removal

It’s a tedious task to remove a watermark manually, which can can take even image editing experts several minutes. Even for a computer it is very difficult to automatically detect and remove a watermark on a single image. However, if watermarks are added in a consistent manner to many images, automatic removal becomes much easier.

In the first step, an algorithm identifies which image structures are repeating in an image collection. If a similar watermark is embedded in many images, the watermark becomes the signal and images become the noise. At that point, a few simple image operations can generate a rough estimation of the watermark pattern.

In the second step, the watermark is separated into its image and opacity components while reconstructing a subset of clean images. The end result is a much more accurate estimation of the watermark pattern, which can then easily be removed from the marked images—no manual photo editing required.

Making More Secure Watermarks

As the vulnerability of current watermarking techniques lies in the consistency in watermarks across image collections, the research team at Google developed a method to introduce inconsistencies when embedding watermarks.

They found that simply changing the watermark position or random changes in its opacity do not improve security by much; however, slightly warping the watermark when embedding it in the image did the trick by producing a watermark that is very similar to the original but leaves very visible artifacts when removed by an algorithm. Estimating the random warp field that was applied to the watermark is simply too difficult for current algorithms.

According to the researchers, there is no guarantee that there will not be a way to break randomized watermarking schemes in the future, but randomized warping will make it fundamentally more difficult to automatically remove watermarks from image collections.

More detail and sample images are available on the Google Research Blog.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

Caltech research team develops lensless camera

23 Jun

Image: Caltech

Smartphone cameras have improved considerably over the past few years but despite innovations such as image stacking and dual cameras with image fusion technology the cameras are still limited by the laws of physics. This becomes particularly evident when looking at the ‘tele’ lenses that have cropped up on some recent high-end smartphones with dual cameras, such as the iPhone 7 Plus or Xiaomi Mi6.

Due to space constraints in the slim smartphone bodies these lenses use smaller sensors and offer considerably slower apertures than their wide angle counterparts which makes them a lot less usable in lower light conditions. However, now it looks like a research team at Caltech could have found a solution to the problem. They have developed an ‘optical phased array’ chip that uses algorithms instead of a lens to focus the incoming light beam. A time delay which can be as short as a quadrillionth of a second, is added to the light captured at different locations on the chip. This allows for modifying focus without a lens.

Professor Ali Hajimiri says the system ‘can switch from a fisheye to a telephoto lens instantaneously – with just a simple adjustment in the way the array receives light.’ The existing 2D, lensless camera array consists of an 8×8 grid with 64 sensors and is capable of capturing a low resolution image of a barcode. The current image results are a long way from current smartphone cameras but at this point the system is only a proof of concept and potential commercial applications are a few years in the future. The team’s next objective is to use larger receivers that are more sensitive and capable of capturing higher-resolution images.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

Adobe Research tackles selfie photography with new AI-powered tech

07 Apr

Adobe Research, the company’s research and development division, has released a look at a new technology that tweaks selfies to improve how they look. The technology, which is presented in a new Adobe video, is designed to improve mobile portrait photography by enabling users to adjust the photo’s perspective, depth of field, and more.

Adobe describes its new technology as ‘the potential future of selfie photography,’ demonstrating how it can be used to replicate a more flattering focal distance, adjust the position of the subject’s head within the image, adjust the depth of field using automatic portrait masking and apply styles found in other portraits, such as images found in a Google Image search.

This technology is powered by Adobe Sensei, an artificial intelligence and deep-learning framework the company introduced at Adobe MAX 2016 last November. The selfie technology isn’t available to consumers at this time, but instead serves to highlight Adobe’s latest developments and to introduce photographers to the kind of tools they may have access to in the future.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

Science City: Futuristic Research Complex to Vitalize Egyptian Desert

12 Sep

[ By WebUrbanist in Architecture & Public & Institutional. ]

science city egypt

A new 125,000-square-foot science complex to be built outside Cairo will combine research, learning and museum facilities in a future-focused structure designed by competition-winning architects.

science side view

Architects Weston Williamson+Partners beat Zaha Hadid Architects to secure the role of lead designer on this forward-thinking Science City complex. Their plans include a planetarium, observation tower, workshop rooms and conference facilities in addition to spaces dedicated to scientific research and development.

science city displays

science center interior

The competition brief called for a future-oriented, state-of-the-art interactive science museum for the new century, and “a set of buildings and spaces that must be inspiring on the outside and motivating and exciting on the inside to visitors and employees alike” – it attracted nearly 500 submissions from architects around the world.

science plan view

science city section

science master plan

The circular footprint is filled with umbrella-shaped protrusions serving to define spaces and paths while providing shelter from the Egyptian heat. While currently located in a semi-remote location, the design is intended to form part of a larger regional master plan for redefining and expanding Egypt’s capital city.

science exterior

futuristic science complex

Of their victory, the architects said: “We are proud to have won. Needless to say that Egypt has a unique cultural heritage, but we were also attracted by the ambition of the project, clearly expressed through the brief. We look forward to developing the design and creating something worthy for Egypt’s future generations.”