Neural « 360Photography

RSS

Posts Tagged ‘Neural’

NVIDIA Research develops a neural network to replace traditional video compression

06 Oct

NVIDIA researchers have demonstrated a new type of video compression technology that replaces the traditional video codec with a neural network to drastically reduce video bandwidth. The technology is presented as a potential solution for streaming video in situations where Internet availability is limited, such as using a webcam to chat with clients while on a slow Internet connection.

The new technology is made possible using NVIDIA Maxine, a cloud-AI video streaming platform for developers. According to the researchers, using AI-based video compression can strip video bandwidth usage down to 1/10th of the bandwidth that would otherwise be used by the common H.264 video codec. For users, this could result in what NVIDIA calls a ‘smoother’ experience that uses up less mobile data.

In a video explaining the technology, researchers demonstrate their AI-based video compression alongside H.264 compression with both videos limited to the same low bandwidth. With the traditional video compression, the resulting low-bandwidth video is very pixelated and blocky, but the AI-compressed video is smooth and relatively clear.

This is made possible by extracting the key facial points on the subject’s face, such as the position of the eyes and mouth, then sending that data to the recipient. The AI technology then reconstructs the subject’s face and animates it in real time using the keypoint data, the end result being very low bandwidth usage compared to the image quality on the receiver’s end.

There are some other advantages to using AI-based compression that exceed the capabilities of traditional video technologies, as well. One example is Free View, a feature in which the AI platform can rotate the subject so that they appear to be facing the recipient even when, in reality, their camera is positioned off to the side and they appear to be staring into the distance.

Likewise, the keypoints extracted from the subject’s face could also be used to apply their movements to other characters, including fully animated characters, expanding beyond the AI-powered filters that have become popular some video apps like Snapchat. Similar technology is already on the market in the form of Apple’s AI-based Animoji.

The use of artificial intelligence to modify videos isn’t new; most major video conferencing apps now include the option of replacing one’s real-life background with a different one, including intelligent AI-based background blurring. However, NVIDIA’s real-time AI-based video compression takes things to a new level by using AI to not only generate the subject in real time, but also modify them in convenient ways, such as aligning their face with a virtual front-facing camera.

The technology could usher in an era of clearer, more consistent video conferencing experiences, particularly for those on slow Internet connections, while using less data than current options. However, the demonstration has also raised concerns that largely mirror ones related to deepfake technologies — namely, the potential for exploiting such technologies to produce inauthentic content.

Artificial intelligence technology is advancing at a clipped rate and, in many cases, can be used to imperceptibly alter videos and images. Work is already underway to exceed those capabilities, however, by fully generating photo-realistic content using AI rather than modifying existing real-world content.

The Allen Institute for AI recently demonstrated the latest evolution in this effort by using both images and text to create a machine learning algorithm that possesses a very basic sense of abstract reasoning, for example. NVIDIA Research has also contributed extensively to this rapidly evolving technology, with past demonstrations including generating landscapes from sketches, generating photo-realistic portraits and even swapping facial expressions between animals.

A number of companies are working to develop counter technologies capable of detecting manipulated content by looking for markers otherwise invisible to the human eye. In 2019, Adobe Research teamed up with UC Berkeley to develop and demonstrate an AI capable of not only identifying portrait manipulations, but also automatically reversing the changes to display the original, unmodified content.

The general public doesn’t yet have access to these types of technologies, however, generally leaving them vulnerable to the manipulated media that permeates social media.

Via: NVIDIA

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

YouTuber upscales classic film to 4K/60p resolution using neural networks

05 Feb

Chances are you’ve seen the famous short film ‘Arrival of a Train at La Ciotat (France),’ by the Lumière Brothers at some point in your life. If not, the original 57-second clip, created in 1895, can be viewed above.

YouTube creator Denis Shiryaev used neural networks to upscale and resound the original black and white clip. His efforts resulted in a 4K/60p clip that is quite astounding. The absence of jerkiness and artifacts makes the arrival of the train that much more impactful and shows just how powerful machine learning has become. Watch Shiryaev’s updated version, below:

You can find more of Shiryaev’s work on his YouTube Channel.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

GANPaint Studio uses neural network to ‘paint’ new elements into images

24 Jan

A team of researchers with IBM Research, MIT CSAIL and MIT-IBM Watson AI Lab has launched a new online tool called GANPaint Studio that utilizes a GAN neural network and semantic brushes to ‘draw’ entirely new elements into existing images. In the case of this particular tool, the elements include grass, clouds, brick, doors, trees, sky and domes.

Unedited before image.

As demonstrated in the images above and below, GANPaint Studio is more of a fun demonstration rather than a serious tool for modifying images. The input images are stripped down to a very low resolution when uploaded; the resulting images are clearly edited, though the neural network is capable of some surprisingly realistic edits.

After adding grass, trees and clouds.

In addition to drawing elements into the images, the tool also features an eraser icon that, when clicked, enables the user to erase elements from the input image. This isn’t the first time we’ve seen a demonstration of a neural network capable of producing realistic elements in an image using a basic ‘drawing’ tool.

In March 2019, for example, NVIDIA Research demonstrated a similar tool it calls GauGAN to generate a photorealistic image from a series of crudely painted marks, each mark made to represent types of elements like water, trees and sky. NVIDIA has published a sizeable body of research on AI and its potential for generating photorealistic images.

As for GANPaint Studio, anyone can access the photo editor here; it comes populated with a selection of preloaded images, but users also have the option of uploading their own image. While using the tool, we found that the images need to be at a fairly low resolution, such as 800 x 500, for the editor to successfully upload the input image.

The MIT and IBM researchers have made their research on the project publicly available [Note: This is a 48MB PDF].

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

DaVinci Resolve 16 has new Neural Engine, native Frame.io integration and more

11 Apr

In addition to the new battery grip for the Pocket Cinema Camera 4K, Blackmagic has announced DaVinci Resolve 16, the latest version of its video editor that brings a massive collection of new and updated features.

The standout feature of DaVinci Resolve 16 is a new cut page designed specifically ‘for editors that need to work quickly and on tight deadlines.’ The updated cut page is an alternate edit page that features a streamlined interface and a new toolset that makes it easier to ingest, process and export footage.

In Blackmagic’s own words, ‘The [new] cut page isn’t about simplification, it’s about removing the things customers don’t need and building powerful, professional tools that help customers work more quickly. And, sometimes, it means borrowing the things that were great about the past and bringing them into the future.’

These new and improved tools include source tape, a new feature that brings all of the clips in a users bin into the viewer as a single long tape so it’s easier to scrub though, select the in/out points and bring the needed footage into the timeline. Another updated tool within the interface is a dual timeline arrangement that makes it possible to see both detailed sections of footage, as well as the whole timeline at once. This makes it easier to get both a macro and micro look at the work, rather than having to zoom in and out constantly.

DaVinci Resolve 16’s Neural Engine at work picking out faces from various clips.

Blackmagic Design has also added its new DaVinci Neural Engine, which uses ‘state of the art deep neural networks and learning, along with artificial intelligence to power new features such as speed warp motion estimation for retiming, super scale for up-scaling footage, auto color and color matching, facial recognition and more.’

The DaVinci Neural Engine is cross-platform and uses the latest GPU technologies to provide improved performance when working on footage and help to streamline the editing process. Blackmagic Design specifically references the DaVinci Neural Engine’s ability to use facial recognition to automatically sort through footage and add individual clips to folders based on who is in the shot.

ResolveFX has also been updated in DaVinci Resolve 16. You can now add vignettes, drop shadows, analog noise/damage, chromatic aberration, video stylization and even remove objects. Blackmagic Design says there have also been improvements to the scalene, beauty, face refinement, blanking fill, warper, dead pixel fixer and colorspace transformation plugins.

Additional features added and improved upon in DaVinci Resolve 16 include new adjustment clips to help add effects and grades to clips in the timeline, a new quick export tool for uploading videos to YouTube and Vimeo from anywhere inside the app and GPU-accelerated scopes to help keep an eye on the technical side of things. Blackmagic has also partnered up with remote collaboration tool Frame.io to add native support in DaVinci Resolve 16. Now, Frame.io is baked right into the software, rather than working as an iteration.

Below is a 25-minute video of Blackmagic Design walking through all of the changes found inside DaVinci Resolve 16:

DaVinci Resolve 16 public beta is available to download from the Blackmagic Design website, where you will also find additional details.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

This neural network turns smartphone snaps into ‘DSLR-quality photos’

31 Oct

Researchers at ETH Zürich have developed an AI-powered system that can turn your measly smartphone snapshots into images that look like they were recorded with a full-blown DSLR… or so they claim.

The project is called ‘DSLR-Quality Photos on Mobile Devices with Deep Convolutional Networks’ and part of the abstract on the project home page reads as follows:

Despite a rapid rise in the quality of built-in smartphone cameras, their physical limitations—small sensor size, compact lenses and the lack of specific hardware—impede them to achieve the quality results of DSLR cameras. In this work we present an end-to-end deep learning approach that bridges this gap by translating ordinary photos into DSLR-quality images.

Of course, the term “DSLR-quality images” could mean many things, but it looks like the software is currently focusing on sharpness, color and tonality. This is in contrast with what smartphone manufacturers tend to refer to as “DSLR-quality images” and what they try to replicate with ‘Portrait’ mode photos: depth-of-field, or rather the lack of it.

To create the software the team started by by training a deep learning system by feeding it photos taken of the same scene using a smartphone camera and a DSLR. This approach worked well but could only improve the quality for the specific smartphone in question. A more sophisticated second version only needs to see two sets of images from different cameras to understand how to apply the image quality from one to the other; in other words: you can feed any photo into the system and apply the image quality of a target camera to it.

The results still need some fine-tuning on occasions—for example, some of the sample shots on display show color casts or a loss of detail after going through the process. However, test images tend to be better exposed and more vibrant. The most obvious improvements can be achieved with smartphone cameras on older or lower-tier devices though.

The scientist hope to eventually use their neural network for modifying the shooting conditions rather than the image quality of the camera. For example, you could turn a photo that was taken on a rainy day into one captured in bright sunshine… for many photographers this might be just a step to far.

If you want to try the current version yourself, you can do so on phancer.com.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

Dream Deep: Trippy Maps Reenvisioned by Google’s Artificial Neural Network

29 Jun

[ By WebUrbanist in Art & Drawing & Digital. ]

FaceApp and similar reality-warping applications are especially fun to use in ways their designers never intended. Along similar lines, Google’s DeepDream (designed for photo manipulation) creates fascinating results using photographs but is even more stunning when applied to representations of cityscapes.

While training DeepDream (a neural network that adapts like a brain to new inputs) to identify, differentiate and understand images, Google researchers discovered it could “over-interpret” results as well. In short: it could start to “read into” images from previous experience, resulting in an array of beautiful (if disturbing) hybrids.

Once it went public, mapmakers were among those intrigued by the possibilities of geo-visualization, turning flat maps into seemingly living landscapes. Tim Waters, a geospatial developer, began taking OpenStreetMap data and running it through the system, generating these strangely psychedelic urban environments.

He discovered that a short run could create fractal and quilting effects, while longer and reiterated processing started to introduce faces and creatures to the mix.

Above: monkeys and frogs seem to emerge from the grid, while a coastal region forms the head of a bear, making the landscape look like a giant bearskin rug. Overall, the effects are quite beautiful, creating a sense of depth and adding character to what would otherwise be fairly generic representations.

[ By WebUrbanist in Art & Drawing & Digital. ]

[ WebUrbanist | Archives | Galleries | Privacy | TOS ]

WebUrbanist

Comments Off

Posted in Creativity

Everypixel Aesthetics uses neural networks to judge your photographs

08 Apr

Designers and image editors often have to browse through large numbers of low-quality photographs before they find the stock image that is most suitable for their purposes. Now, a new algorithm has been created to filter images based on their aesthetic value and get rid of the junk before it clogs up your search results.

Everypixel uses neural networks for ranking stock images and for this purpose has trained the algorithms to judge the aesthetic value of a stock image in the same way as a human would do.

Everypixel’s CEO Dmitry Shironosov said: “Designers, editors and experienced stock photographers helped us generate a training dataset with 946,894 positive and negative patterns. We wanted to create a technology that can measure not only aesthetics of stock images, but their commercial potential as well. This is the main difference between our smart filter and other solutions that exist today.”

The neural network is capable of estimating the visual quality of an image and applies a score to every file which, if working properly, could save many man hours of human image curation. The algorithm is currently in beta stage but you can already test it with your own images on Everypixel. We’re not so sure about the scoring, but the system already looks pretty good at assigning correct keywords. How did your images do? Let us know in the comments.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

Neural network converts Game Boy Camera images into color photos

21 Feb

We’ve seen a lot of research lately that uses neural networks to upsample low resolution images and the results have been impressive – even a little creepy. Google recently showcased a system that can turn a low resolution 8×8 input image into a 32×32 sample that’s remarkably close to the original image. Inspired by recent breakthroughs, research engineer Roland Meertens found another application for neural networks – one that’s highly relevant to our interests. He created an application that turns low-res, monochrome Game Boy Camera images into photorealistic color images.

Original images in the center, Game Boy-ified images on the left and image generated by neural network on the right

A network must be trained, and training means feeding it input images. To create a training data set, Meertens gave some ‘real life images’ a Game Boy Camera treatment by re-creating them in four shades of black. By comparing the Game Boy-ified images with the originals, the network is ‘taught’ how to convert the images to color. With the network trained and ready, Meertens began testing it on celebrity photos as well as images from the Game Boy Camera (including the game’s mysterious character at the top of the page).

Finally, Meertens uses the application on an image taken with the Game Boy Camera. Naturally, it should be a selfie, as it is here. If you have all of the necessary components, taking a photo with the Game Boy camera is easy. Getting it onto your computer is another story. Lacking a specialized cable, Meertens did his best to photograph the Game Boy screen. As a result the lighting is slightly uneven, which affects the output from the network, but the re-creation is still pretty darn cool. Our hats are off to him.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

No dual-cam? No problem: Patch app for iOS uses neural networks to create fake bokeh images

11 Nov

Most dual-cam equipped smartphones offer a ‘fake bokeh’ feature. Thanks to the slightly offset position of their two lenses, cameras in devices like the Apple iPhone 7 Plus, Huawei P9 or LG G5, can distinguish between objects in the foreground and background of an image. By applying digital blur to the latter they can simulate effects of shallow depth-of-field you would typically achieve with a DSLR and fast lens.

If your phone just has one camera, there are still a few pure software solutions out there to achieve the same effect. The Patch app for iOS is the latest and uses neural networking to identify the foreground subject in an image and isolate it from the background. If the scene is too complex for the algorithms to work automatically, there is also a manual selection tool that can be used to optimize the results. You can paint in areas that should be sharp, and remove areas that should be blurred. A zooming function allows for greater precision in this task.

Once the selection is finalized users can choose from 5 different blur strengths to generate the desired effect. Patch does not have any particular camera hardware requirements and therefore works with most iOS devices. If you want to try the app you can download a free version that will leave a watermark on your images from the Apple App Store. A $ 1 in-app purchase will get you an upgrade to the watermark-free version.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

Google uses neural networks to improve image compression

27 Aug

A research team at Google has developed a way to use neural networks to compress image files in a more efficient way than current methods, such as the JPEG standard. The team built an artificial intelligence system using Google’s open source TensorFlow machine learning system, and then used 6 million random reference photos from the internet that had been compressed using conventional methods to train it.

The images were split into small pieces measuring 32 x 32 pixels each. The system then analyzed the 100 pieces with the least efficient compression; the idea being that it could learn from looking at the most complex areas of an image, making compression of less complex sections much easier.

After the initial training process the AI system is then able to predict how the image would look like after compression and then generates that image. What makes this method really stand out from others is that the network can intelligently decide which is the best way to compress individual areas of a given photo for the best overall result. The method still needs some work, as final results can sometimes look unpleasant to the human eye and the system are not yet capable of testing for this. Nevertheless, the project looks like an important step into the right direction and if the algorithms can be further refined you might soon be able to save even more images on your memory card or built-in device storage.

Articles: Digital Photography Review (dpreview.com)

Comments Off

Posted in Uncategorized

360Photography

Posts Tagged ‘Neural’

NVIDIA Research develops a neural network to replace traditional video compression

YouTuber upscales classic film to 4K/60p resolution using neural networks

GANPaint Studio uses neural network to ‘paint’ new elements into images

DaVinci Resolve 16 has new Neural Engine, native Frame.io integration and more

This neural network turns smartphone snaps into ‘DSLR-quality photos’

Dream Deep: Trippy Maps Reenvisioned by Google’s Artificial Neural Network

Everypixel Aesthetics uses neural networks to judge your photographs

Neural network converts Game Boy Camera images into color photos

No dual-cam? No problem: Patch app for iOS uses neural networks to create fake bokeh images

Google uses neural networks to improve image compression

Pages

Archives

Categories