RSS
 

Posts Tagged ‘generates’

New DAIN algorithm generates near-perfect slow-motion videos from ordinary footage

09 Sep

Researchers with Google, UC Merced and Shanghai Jiao Tong University have detailed the development of DAIN, a depth-aware video frame interpolation algorithm that can seamlessly generate slow-motion videos from existing content without introducing excessive noise and unwanted artifacts. The algorithm has been demonstrated in a number of videos, including historical footage boosted to 4K/60fps.

Rapidly advancing technologies have paved the way for high-resolution displays and videos; the result is a mass of lower-resolution content made for older display and video technologies that look increasingly poor on modern hardware. Remastering this content to a higher resolution and frame rate will improve the viewing experience, but would typically be a costly undertaking reserved only for the most popular media.

Artificial intelligence is a promising solution for updating older video content as evidenced by the growing number of fan-remastered movies and TV shows. Key to these efforts are algorithms trained to upscale and, when necessary, repair the individual frames of videos, which are recompiled into a higher-resolution ‘remaster.’

The newly detailed DAIN algorithm is different — rather than upscaling and repairing the individual frames in a video, this AI tool works by generating new frames and slotting them between the original frames, increasing the video’s FPS for smoother and, depending on how many frames are generated, slower-motion content.

This is a process called motion (video frame) interpolation, and it typically causes a drop in quality by adding unwanted noise and artifacts to the final videos. The DAIN algorithm presents a solution to this problem, offering motion interpolation to boost frames-per-second up to 480fps without introducing any readily noticeable artifacts.

The resulting content is high-quality and nearly visually identical to the source footage, but with the added smoothness that comes with increasing the frames-per-second to 60fps. In addition, DAIN has been demonstrated as capable of transforming ordinary 30/60fps footage into smooth slow-motion videos without choppiness or decreased quality.

According to the researchers, DAIN is ‘compact, efficient, and fully differentiable,’ offering results that perform ‘favorably against state-of-the-art frame interpolation methods on a wide variety of datasets.’ The technology has many potential uses, including recovering lost frames, improving content to be more visually appealing for viewers, generating slow-motion from regular footage and more.

Such technology is arguably necessary for preserving aging media in a useful way, making it possible for new generations of people to experience historical footage, old TV shows and movies, home videos and similar content using modern high-resolution displays. As well, the technology could be useful for content creators of all sorts, enabling them to salvage the footage they already have, improve the quality of old clips for use in documentaries and similar things.

The researchers explain on their project website:

Starting from the birth of photographing in the 18-th centuries, videos became important media to keep vivid memories of their age being captured. And it’s shown in varying forms including movies, animations, and vlogs. However, due to the limit of video technologies including sensor density, storage and compression, quite a lot of video contents in the past centuries remain at low quality.

Among those important metrics for video quality, the most important one is the temporal resolution measured in frame-per-second or fps for short. Higher-frame-rate videos bring about more immersive visual experience to users so that the reality of the captured content is perceived. Therefore, the demand to improve the low-frame-rate videos, particularly the 12fps old films, 5~12fps animations, pixel-arts and stop motions, 25~30 fps movies, 30fps video games, becomes more and more urgent.

The public can view more examples of videos updated using the DAIN algorithm by checking out the related collection playlist on YouTube. As well, the full study is available in PDF form on the Arxiv website here.

Articles: Digital Photography Review (dpreview.com)

 
Comments Off on New DAIN algorithm generates near-perfect slow-motion videos from ordinary footage

Posted in Uncategorized

 

NVIDIA researchers create AI that generates photo-realistic portraits

19 Dec

NVIDIA researchers have published a new paper detailing their latest artificial intelligence work, which involves generating photo-realistic portraits of humans that are indistinguishable from images of real people. The technology revolves around an alternative generator architecture for generative adversarial networks (GANs) that utilizes style transfer for producing the final result.

Though GANs have improved substantially in only a few years, the researchers say in their paper that the generators ‘continue to operate as black boxes, and despite recent efforts, the understanding of various aspects of the image synthesis process, e.g., the origin of stochastic features, is still lacking.’ That’s where the newly developed alternative architecture comes in.

The team’s style-based architecture enables GANs to generate new images based on photos of real subjects, but with a twist: their generator learns to distinguish between separate elements in the images on its own. In the video above, NVIDIA’s researchers demonstrate this technology by generating portraits based on separate elements from images of real people.

“Our generator thinks of an image as a collection of ‘styles,’ where each style controls the effects at a particular scale,” the team explains.

Image elements are split into three style categories: “Coarse,” “Middle,” and “Fine.” In terms of portraits, these categories include elements like facial features, hair, colors, eyes, the subject’s face shape, and more. The system is also able to target inconsequential variations, including elements like texture and hair curls/direction.

The video above demonstrates changes involving inconsequential variation on non-portrait images, which includes generating different patterns on a blanket, altering the hair on a cat, and subtly changing the background behind a car. The style-transfer GANs offer superior results to traditional GAN generator architecture, the researchers conclude, with the photo-realistic results underscoring their assessment.

The latest work further refines a technology that has been growing rapidly over only a few years. Though GANs have been used in the past to generate portraits, the results were far from photo-realistic. It’s possible that technology like this could one day be offered as a consumer or enterprise product for generating on-demand life-like images.

Articles: Digital Photography Review (dpreview.com)

 
Comments Off on NVIDIA researchers create AI that generates photo-realistic portraits

Posted in Uncategorized

 

New technology alters perspective in selfies, generates 3D images, and more

31 Jul

A team of researchers from Princeton University and Adobe Research have detailed a new project in which they use a 3D computer model of a head and a virtual ‘full perspective’ camera to manipulate the perspective of a single portrait. The manipulations simulate various shooting distances and the warps typically seen at those depths, potentially allowing software adjustments that create selfies with corrected perspective distortion.

A demo system (currently in beta) on lead researcher Ohad Fried’s website allows you to upload your own images to explore the technology.

The front-facing lenses found in smartphones cameras are often wide-angle, fixed focal length, to make them as flexible as possible, but the close-up nature of selfies tends to show distortions such as large noses or sloping foreheads. Interestingly, these distortions can change how the individuals are perceived; the subjects in portraits taken at close distances are often described in ways that include ‘approachable’ and ‘peaceful’ while subjects in portraits taken at longer distances are more often described as ‘smart,’ ‘strong,’ and ‘attractive.’

While it might be beneficial to take selfies at longer distances and longer focal lengths to eliminate the distortion, there is no practical way to do so with present phone technology. This newly developed technology could change that, however, with the researchers explaining: ‘our framework allows one to simulate a distant camera when the original shot was a selfie, and vice versa, in order to achieve various artistic goals.’

The researchers based their method on existing approaches to manipulating images, including the type of technology used in face-swapping apps. The key difference was using a ‘full perspective’ virtual camera model rather than a more simplistic, ‘weak perspective’ model, enabling them to compensate for the wider range of perspective adjustments needed for portraits taken at very close distances. This new method is able to estimate the camera distance and edit the perceived camera distance. Its modeling of depth also allows slight changes in the position of the virtual camera, allowing the photos to be slightly ‘re-posed’.

The technology promises than just correcting selfie perspective. The ability to slightly correct perspective and map facial features to a 3D model allows the creation of stereo pairs of images (3D anaglyphs) from a single image, or could make it possible to animate changes in facial expressions.

Articles: Digital Photography Review (dpreview.com)

 
Comments Off on New technology alters perspective in selfies, generates 3D images, and more

Posted in Uncategorized