RSS
 

Posts Tagged ‘Researchers’

Researchers use iPhone 5 camera and LEGO to create affordable high-resolution microscope

02 Jul

There are millions of old and outdated iPhones collecting dust. Researchers in Germany have found a way to turn some of those old iPhones, specifically an iPhone 5 camera module, into affordable microscopes for young students. Using LEGO, an iPhone 5 camera, LED lighting and a modern smartphone, students can build their own microscope.

Researchers Bart E. Vos, Emil Betz Blesa and Timo Betz from Georg August University Göttingen and Munster University in Germany set out to build a high-resolution microscope that wasn’t prohibitively expensive. Toy microscopes aren’t very effective, and specialized microscopes cost a lot of money, limiting their accessibility.

The researchers said, ‘Our aim is to introduce a microscope to individual students in a classroom setting, both as a scientific tool to access the micro-world and to facilitate the understanding of fundamental principles of the optical components of a microscope in a playful and motivating, yet precise approach. By basing the design on LEGO, we aim to make the microscope modular, cheap, and inspiring.’

‘Design of the LEGO microscope. (a, b) A photograph and a schematic representation of the microscope, (c) the LED that illuminates the sample from below, (d) the threaded system that adjusts the focus of the microscope by moving the objective, (e) 2 objectives containing a replacement smartphone lens with a 3.85-mm focal distance (left) and a glass lens with a 26.5-mm focal distance (right), (f) the second lens consisting of 2 acrylic lenses in its holder just below the eyepiece, (g) a smartphone used as a camera by adapting the eyepiece.’ Credit: Bart E. Vos, Emil Betz Blesa and Timo Betz

The researchers used an iPhone 5 camera module, smartphone and LEGO housing to craft a high-resolution microscope. Many people already have LEGO pieces around, and iPhone 5 lenses are quite cheap to come by. The researchers found one for under $ 5. The project’s full price, without including the cost of a modern smartphone, is €102 (about $ 120 USD). There’s a bit more to it, but it’s straightforward and inexpensive. Documentation for building your own microscope is available for free.

‘Schematic overview of the light path in the microscope. The object (here depicted as an arrow) forms an inverted intermediate image in the focus of the second lens. The second lens then sends collimated light to the observer.’ Credit: Bart E. Vos, Emil Betz Blesa and Timo Betz

The hope is that the LEGO microscope will make science more accessible to children worldwide. Every child deserves the opportunity to learn about our world, including the parts of it we can’t see with the naked eye. ‘An understanding of science is crucial for decision-making and brings many benefits in everyday life, such as problem-solving and creativity,’ said Professor Timo Betz, University of Göttingen. ‘Yet we find that many people, even politicians, feel excluded or do not have the opportunities to engage in scientific or critical thinking. We wanted to find a way to nurture natural curiosity, help people grasp fundamental principles and see the potential of science.’

‘Examples of experiments conducted with the LEGO microscope. (a) Image of a sodium chloride crystal. (b) Time lapse of an osmotic shock in red onion cells. After approximately 30 s, a 1 M NaCl solution is flowed in. Subsequently, water leaves the cells, causing the cell membranes to detach from the cell walls. After approximately 5 min, distilled water is flowed in, washing away the 1 M NaCl solution, and the cells return to their original volume. (c) Time lapse of the movement of an Artemia shrimp in water. (d) Time lapse of the movement of 2 water fleas in water. The scale bars in panels a, b, and d are 100 lm.’ Credit: Bart E. Vos, Emil Betz Blesa and Timo Betz

In addition to providing the plans for free, Vos, Blesa and Betz also published a paper about the microscope project.

Articles: Digital Photography Review (dpreview.com)

 
Comments Off on Researchers use iPhone 5 camera and LEGO to create affordable high-resolution microscope

Posted in Uncategorized

 

Researchers propose ‘spaceplates’ to miniaturize lenses by reducing air gaps

17 Jun
The addition of the spaceplate reduces the distance needed between the lens element and the sensor, thus allowing smaller lenses

A lot of work has been done using high refractive index glasses, diffraction grating systems and lens element design to reduce the size of camera lenses, but a group of researchers are now targeting the air-space between those elements in a bid to create miniaturized optical systems. The team from the University of Ottawa proposes inserting what they call ‘spaceplates’ into a lens construction to alter the optical path in such a way that the gaps between elements in the lens can be reduced. They further propose that when combined with metalenses these spaceplates could, in theory, allow optical systems that are almost flat and extremely thin.

This shows how the research team propose the spaceplate idea could make regular lenses smaller, and replaced when the spaceplate is combined with a metalens

In any lens it is the area reserved for air – the gaps between the elements – that takes up the most space. These gaps of course are carefully calculated and are key to directing the path of light as it passes through from the front element to the camera’s sensor. The idea here is to compress those gaps using multiple layers of metasurfaces that provide negative refractive indexes to shorten the light path between one element and the next. In photographic and telescope optics mirror lenses aim to achieve a similar end, not so much by shortening the light path but by allowing the same distance to be traveled inside a shorter-than-usual lens barrel.

Metasurfaces are materials that alter the path of light not by using bulbous glass or plastic elements but by tiny structures within their make up. As light passes through grids, nets and grates within the material redirect the light, altering its path. The grating system in Canon’s DO lenses works in a broadly similar way bit on a different scale.

Trails using oil between the lens element and the spaceplate showed that the same area of the subject, a painting in this case, could be rendered in-focus with less distance between the lens and the sensor when a spaceplate was used.

The spaceplate idea is still very much at the concept stage, and trails conducted have used liquids and vacuums instead of air. They have also produced relatively small improvements, but at the same time the construction of the metasurface layers of the spaceplates has been kept relatively simple. So far the team has achieved a compression factor of R=5, and say that if they can achieve a factor of R=40 by combining multiple layers of metasurface materials to a thickness of 100µm they could reduce the air space in a typical smartphone camera lens from 1mm to 0.1mm.

Although the technology is most likely to be employed in industrial processes before consumer products, the idea does offer potential for interchangeable lens system cameras too. The team has demonstrated that the spaceplate does not affect focal length, works with all visible wavelengths and offers high transmission efficiency. Scaling up to spaceplates with more metasurfaces should be relatively easy as manufacturing processes are already in use.

Don’t expect to see spaceplates in camera lenses anytime soon, but it certainly could be something we see in the future in other products, such as projection lenses in AR/VR and holographic headsets. For more information you can read the full paper on the Nature website. Warning: it’s 6700 words long, isn’t easy reading and contains no jokes.

Articles: Digital Photography Review (dpreview.com)

 
Comments Off on Researchers propose ‘spaceplates’ to miniaturize lenses by reducing air gaps

Posted in Uncategorized

 

Researchers create 100 billion FPS 3D camera with lens that mimics human eyes

28 Oct

A study recently published in Nature Communications details the creation of an ultra-fast 3D camera capable of recording at 100 billion frames per second. The development comes from Lihong Wang and his team at Caltech, where the researchers developed this new camera using the same foundational technology found in Wang’s previous 70 trillion frames per second project.

The newly detailed camera produces what the researchers call ‘single-shot stereo-polarimetric compressed ultrafast photography’ (SP-CUP), a technology that records video at insanely fast speeds in three dimensions. This is made possible, in part, by creating the camera to perceive the world in a way similar to how a human sees — with two eyes, or in the case of the camera, with a halved lens that simulates looking at the world with two eyes.

The result is a camera that records video at insanely fast speeds in three dimensions. The technology is able to capture ‘non-repeatable 5D … evolving phenomena at picosecond temporal resolution,’ according to the study, referring to space, time of arrival, and angle of linear polarization. The study goes on to explain:

Disruptively advancing existing CUP techniques in imaging capability, SP-CUP enables simultaneous and efficient ultrafast recording of polarization in three-dimensional space. Compared with available single-shot ultrafast imaging techniques, SP-CUP has prominent advantages in light throughput, sequence depth, as well as spatiotemporal resolution and scalability in high-dimensional imaging.

Wang and his lab first detailed the 70 trillion frames per second camera back in May, explaining that such speeds were capable of capturing the fluorescent decay from molecules and waves of light as they traveled.

That particular camera technology was called compressed ultrafast spectral photography (CUSP), and it followed Wang’s past work on similar technologies, including the phase-sensitive compressed ultrafast photography (pCUP) device, Caltech had explained in a release.

With the newly detailed SP-CUP technology, the camera captures stereo imagery — 10 billion images in the blink of an eye — using a single lens that has been halved in order to capture two different slightly offset channels of the subject. This is similar to how the human eye works, enabling humans to perceive depth. The image data can be processed to create 3D content, which itself exceeds the capabilities of the human eye by including data on the polarization of light.

The sum total of this new photography technology opens the door for various scientific applications, including research in the field of physics. In particular, Wang sees the potential use of this camera in exploring the mystery of sonoluminescence, a phenomenon in which sound waves produce small bubbles in liquids that, when they collapse, produce tiny bursts of light.

Wang explained:

Some people consider this one of the greatest mysteries in physics. When a bubble collapses, its interior reaches such a high temperature that it generates light. The process that makes this happen is very mysterious because it all happens so fast, and we’re wondering if our camera can help us figure it out.

The study titled ‘Single-shot stereo-polarimetric compressed ultrafast photography for light-speed observation of high-dimensional optical transients with picosecond resolution’ is available in Nature Communications.

Articles: Digital Photography Review (dpreview.com)

 
Comments Off on Researchers create 100 billion FPS 3D camera with lens that mimics human eyes

Posted in Uncategorized

 

Researchers teach an AI to generate logical images based on text captions

30 Sep

The Allen Institute for AI (AI2) created by Paul Allen, best known as co-founder of Microsoft, has published new research on a type of artificial intelligence that is able to generate basic (though obviously nonsensical) images based on a concept presented to the machine as a caption. The technology hints at an evolution in machine learning that may pave the way for smarter, more capable AI.

The research institute’s newly published study, which was recent highlighted by MIT, builds upon the technology demonstrated by OpenAI with its GPT-3 system. With GPT-3, the machine learning algorithm was trained using vast amounts of text-based data, something that itself builds upon the masking technique introduced by Google’s BERT.

Put simply, BERT’s masking technique trains machine learning algorithms by presenting natural language sentences that have a word missing, thus requiring the machine to replace the word. Training the AI in this way teaches it to recognize language patterns and word usage, the result being a machine that can fairly effectively understand natural language and interpret its meaning.

Building upon this, the training evolved to include an image with a caption that has a missing word, such as an image of an animal with a caption describing the animal and the environment — only the word for the animal was missing, forcing the AI to figure out the right answer based on the sentence and related image. This taught the machine to recognize the patterns in how visual content related to the words in the captions.

This is where the AI2 research comes in, with the study posing the question: ‘Do vision-and-language BERT models know how to paint?‘

Experts with the research institute build upon the visual-text technique described above to teach AI how to generate images based on its understanding of text captions. To make this possible, the researchers introduced a twist on the masking technique, this time masking certain parts of images paired with captions to train a model called X-LXMERT, an extension of the LXMERT model family that uses multiple encoders to learn connections between language and visual data.

The researchers explain in the study [PDF]:

Interestingly, our analysis leads us to the conclusion that LXMERT in its current form does not possess the ability to paint – it produces images that have little resemblance to natural images …

We introduce X-LXMERT that builds upon LXMERT and enables it to effectively perform discriminative as well as generative tasks … When coupled with our proposed image generator, X-LXMERT is able to generate rich imagery that is semantically consistent with the input captions. Importantly, X-LXMERT’s image generation capabilities rival state-of-the-art image generation models (designed only for generation), while its question-answering capabilities show little degradation compared to LXMERT.

By adding the visual masking technique, the machine had to learn to predict what parts of the images were masked based on the captions, slowly teaching the machine to understand the logical and conceptual framework of the visual world in addition to connecting visual data with language. For example, a clock tower located in a town is likely surrounded by smaller buildings, something a human can infer based on the text description.

An AI-generated image based on the caption, ‘A large painted clock tower in the middle of town.’

Using this visual masking technique, the AI2 researchers were able to impart the same general understanding to a machine given the caption, ‘A large clock tower in the middle of a town.’ Though the resulting image (above) isn’t realistic and wouldn’t be mistaken for an actual photo, it does demonstrate the machine’s general understanding of the meaning of the phrase and the type of elements that may be found in a real-world clocktower setting.

The images demonstrate the machine’s ability to understand both the visual world and written text and to make logical assumptions based on the limited data provided. This mirrors the way a human understands the world and written text describing it.

For example, a human, when given a caption, could sketch a concept drawing that presents a logical interpretation of how the captioned scene may look in the real world, such as computer monitors likely sitting on a desk, a skier likely being on snow and bicycles likely being located on pavement.

This development in AI research represents a type of simple, child-like abstract thinking that hints at a future in which machines may be capable of far more sophisticated understandings of the world and, perhaps, any other concepts they are trained to understand as related to each other. The next step in this evolution is likely an improved ability to generate images, resulting in more realistic content.

Using artificial intelligence to generate photo-realistic images is already a thing, though generating highly specific photo-realistic images based on a text description is, as shown above, still a work in progress. Machine learning technology has also been used to demonstrate other potential applications for AI, such as a study Google published last month that demonstrates using crowdsourced 2D images to generate high-quality 3D models of popular structures.

Articles: Digital Photography Review (dpreview.com)

 
Comments Off on Researchers teach an AI to generate logical images based on text captions

Posted in Uncategorized

 

MIT and UMass researchers develop world’s first flat ultra-wide-angle fisheye lens

21 Sep

Researchers with the University of Massachusetts at Lowell and MIT have developed a new type of fisheye lens that is flat and crafted from a single piece of glass. The lens is round, according to the researchers, and it is capable of capturing sharp 180-degree panoramas. This is the first flat fisheye lens made from a single piece of glass, which measures 1mm thick.

Ordinary spherical fisheye lenses are made from multiple pieces of glass designed to bend the light in such a way that it produces circular wide-angle images. The newly developed flat lens instead captures wide-angle panoramas by utilizing ‘tiny structures’ that scatter light in place of the curved glass elements in more costly spherical fisheye lenses.

The version of the lense introduced by the researchers is made for infrared photography, but the team says that it could be modified for use as a regular visible spectrum lens, as well. The flat design is ultimately more compact and less expensive to produce than spherical multi-element lenses.

The researchers envision a variety of uses for their lens design beyond interchangeable lenses. The thin, flat nature of the design would make it possible to implement the fisheye into smartphones, for example, eliminating the need to use a third-party lens add-on. Similar implementation could be used with laptops, VR headsets and even devices like medical imaging equipment.

MIT associate professor Juejun Hu, one of the researchers on the project, explained:

This design comes as somewhat of a surprise, because some have thought it would be impossible to make a metalens with an ultra-wide-field view. The fact that this can actually realize fisheye images is completely outside expectation. This isn’t just light-bending — it’s mind-bending.

Metalens refers to a flat lens that has tiny structures for focusing light. While wide-angle metalenses aren’t new, the researchers note that a single piece of glass without any extra optics have been limited to 60-degrees. The newly published study details how the team got around these restrictions to develop an ultra-wide-angle lens capable of capturing 180-degree panoramas without extra components.

This 180-degree fisheye metalens features a single piece of transparent glass made from calcium fluoride with a lead telluride film on one side. A pattern of ‘optical structures’ called meta-atoms was carved into the film using lithographic techniques, the result of which were many ‘nanoscale geometries’ used to bend the light in specific, precise ways.

The carved structures can introduce phase delays into the scattering of the light — depending on their shape — to imitate the natural phase delays produced by the curved glass elements in spherical fisheye lenses. The light passes from the carved structures on the back of the lens through an optical aperture on the front of the lens.

Study co-author Mikhail Shalaginov said:

When light comes in through this aperture, it will refract at the first surface of the glass, and then will get angularly dispersed. The light will then hit different parts of the backside, from different and yet continuous angles. As long as you design the back side properly, you can be sure to achieve high-quality imaging across the entire panoramic view.

The study was partially funded by DARPA through its EXTREME program, which tasks experts with developing optical tools ‘to enable new functionality and/or vastly improve size, weight, and power characteristics of traditional optical systems.’ The agency goes on to explain that EXTREME will ‘explore this optical design space and aims to understand the trade-offs, and harness the possibilities, afforded’ by Engineered Optical Materials (EnMats).

Via: MIT

Articles: Digital Photography Review (dpreview.com)

 
Comments Off on MIT and UMass researchers develop world’s first flat ultra-wide-angle fisheye lens

Posted in Uncategorized

 

Researchers capture 3,200MP image using future telescope camera

10 Sep

Researchers at the Department of Energy’s SLAC National Accelerator Laboratory have captured 3,200MP images, the largest photos ever captured in a single shot. The camera, an array that contains 189 individual image sensors, will become the future camera of the Legacy Survey of Space and Time (LSST) telescope at the Vera C. Rubin Observatory in Chile. The camera will be used to help shed light on some of the most intriguing mysteries of the universe, including dark matter and dark energy.

The 189 image sensors are charge-coupled devices (CCD) and each capture a 16MP image. To build the image sensor array, nine CCDs and supporting electronics were assembled into square units, called science rafts, by the Department of Energy’s Brookhaven National Laboratory and then shipped to SLAC. Then the team at SLAC inserted 21 of these square units into a grid to hold them in place.

The completion of the image sensor array and focal plane earlier this year took six months and proved to be a difficult task. In order to maximize the imaging area of the array, the gaps between individual image sensors are less than five human hairs wide. If the sensors touch each other during the process, they could easily break. Damaging a sensor or raft would be costly, as the rafts cost up to $ 3M USD a piece. SLAC mechanical engineer Hannah Pollek said of the assembly process, ‘The combination of high stakes and tight tolerances made this project very challenging. But with a versatile team we pretty much nailed it.’

The focal plane features impressive specifications beyond even the 3.2 billion total pixels. The pixels themselves are about 10 microns wide and the focal plane itself has been constructed to exacting standards. The focal plane is nearly perfectly flat, varying by ‘no more than a tenth of the width of a human hair’ across its more than two feet of width. The optics through which light will reach the image sensor array is designed to allow the sensors to identify objects 100 million times dimmer than what the human eye can see. This is equivalent to being able to see a lit candle from thousands of miles away.

The images produced by the 3,200MP camera are so large that you would need nearly 400 4K UHD televisions to display a single image at its full size. The resolving power of the camera would allow you to spot a golf ball from about 15 miles away.

As mentioned, the camera will be installed at the Vera C. Rubin Observatory in Chile. Once it has been installed, it will capture panoramic images of the southern sky every few nights for 10 years.

‘The complete focal plane of the future LSST Camera is more than 2 feet wide and contains 189 individual sensors that will produce 3,200-megapixel images. Crews at SLAC have now taken the first images with it. Explore them in full resolution using the links at the bottom of the press release. (Jacqueline Orrell/SLAC National Accelerator Laboratory)’ Image and caption credit: SLAC

Steven Ritz, project scientists for the LSST Camera at the University of California, Santa Cruz, said, ‘These specifications are just astounding. These unique features will enable the Rubin Observatory’s ambitious science program.’ Over the course of a decade, the camera will capture images of about 20 billion galaxies. Ritz continues, ‘These data will improve our knowledge of how galaxies have evolved over time and will let us test our models of dark matter and dark energy more deeply and precisely than ever.’

Before the focal plane can be used within the Rubin Observatory’s program, it needs to be rigorously tested. This includes capture images of a variety of objects, including a head of Romanesco broccoli. In order to operate normally, the sensors must be cooled to negative 150° Fahrenheit. Without a fully assembled camera, the team at SLAC used a 150-micron pinhole to project images onto the focal plane.

‘Taking the first 3,200-megapixel images was an important first test for the focal plane. To do so without a fully assembled camera, the SLAC team used a 150-micron pinhole to project images onto the focal plane. Left: Schematic of a pinhole projector that projects images of a Romanesco’s detailed texture onto the focal plane. Right: SLAC’s Yousuke Utsumi and Aaron Roodman remove the pinhole projector from the cryostat assembly after projecting the first images onto the focal plane. (Greg Stewart/Jacqueline Orrell/SLAC National Accelerator Laboratory)’ Image and caption credit: SLAC

SLAC’s Aaron Roodman is the scientist responsible for building and testing the LSST Camera. Of the successful test images, he says, ‘Taking these images is a major accomplishment. With the tight specifications, we really pushed the limits of what’s possible to take advantage of every square millimeter of the focal plane and maximize the science we can do with it.’

Despite the successful tests, there is much more work to do. Over the next few months, the team will insert the cryostat used to reduce the temperature of the image sensors along with the focal plane into the camera body and add lenses, including the world’s largest optical lens. The team will then affix a shutter and a filter exchange system so that the camera can be used to capture the night sky in different colors. The team anticipates the SUV-sized camera to be ready for final testing in mid-2021 before it begins its final journey to Chile.

‘Over the next few months, the LSST Camera team will integrate the remaining camera components, including the lenses, a shutter and a filter exchange system. By mid-2021, the SUV-sized camera will be ready for final testing. (Chris Smith/SLAC National Accelerator Laboratory)’ Image and caption credit: SLAC

JoAnne Hewett, chief research officer at SLAC and associate lab director for fundamental physics, says, ‘Nearing completion of the camera is very exciting…It’s a milestone that brings us a big step closer to exploring fundamental questions about the universe in ways we haven’t been able to before.’

As one would expect, we are unable to display 3,200MP images here on the site. However, SLAC has five full-size images taken with the focal plane of the LSST camera which you can view at the links below:

• Head of Romanesco broccoli

• Photo of the Flammarion engraving

• Photo of Vera Rubin, courtesy of the Carnegie Institution for Science, where Vera Rubin spent her career as a scientist

• Collage of LSST Camera team photos

• Collage of logos of institutions involved in the LSST Camera project

Articles: Digital Photography Review (dpreview.com)

 
Comments Off on Researchers capture 3,200MP image using future telescope camera

Posted in Uncategorized

 

Google and UC Berkeley researchers create AI that can remove shadows from images

25 Aug

Researchers with the University of California Berkeley and Google Research have published a new paper detailing an AI that can remove unwanted shadows from images. The algorithm focuses on two different types of shadows — ones from external objects and ones naturally resulting from facial features — and works to either remove or soften them in order to maintain a natural appearance.

Whereas professional images are often taken in a studio with proper lighting, the average snapshot of a person is taken ‘in the wild’ where lighting conditions may be harsh, causing dark shadows that obscure parts of the subject’s face while other parts are covered with excessive highlights.

The newly developed AI is designed to address this problem by targeting those unwanted shadows and highlights, removing and softening them until a clearer subject remains. The researchers say their tool works in a ‘realistic and controllable way,’ and it could prove useful for more than just images captured in casual settings.

Professionals could, for example, use a tool like this to salvage images taken in outdoor environments where it was impossible to control the lighting, such as wedding images taken outdoors under a bright noon sun. In their paper, the researchers explain:

In this work, we attempt to provide some of the control over lighting that professional photographers have in studio environments to casual photographers in unconstrained environments … Given just a single image of a human subject taken in an unknown and unconstrained environment, our complete system is able to remove unwanted foreign shadows, soften harsh facial shadows, and balance the image’s lighting ratio to produce a flattering and realistic portrait image.

This project is designed to target three specific elements in these photographs: foreign shadows from external objects, facial shadows caused by one’s natural facial features and lighting ratios between the lightest and darkest parts of the subject’s face. Two different machine learning models are used to target these elements, one to remove foreign shadows and the other to soften facial shadows alongside lighting ratio adjustments.

The team evaluated their two machine learning models using both ‘in the wild’ and synthetic image datasets. The results are compared to existing state-of-the-art technologies that perform the same functions. ‘Our complete model clearly outperforms the others,’ the researchers note in the study, highlighting their system’s ability in a selection of processed sample images.

In addition to using the technology to adjust images, the study explains that this method can be tapped as a way to ‘preprocess’ images for other image-modifying algorithms, such as portrait relighting tools. The researchers explain:

Though often effective, these portrait relighting techniques sometimes produce suboptimal renderings when presented with input images that contain foreign shadows or harsh facial shadows. Our technique can improve a portrait relighting solution: our model can be used to remove these unwanted shadowing effects, producing a rendering that can then be used as input to a portrait relighting solution, resulting in an improved final rendering.

The system isn’t without limitations, however, particularly if the foreign shadows are presented with ‘many finely-detailed structures,’ some residue of which may remain even after the images are processed. As well, and due to the way the system works, some bilaterally symmetric shadows may not be removed from subjects,

In addition, softening the facial shadows using this technique may, at times, result in a soft, diffused appearance due to excessive smoothing of some fine details that should remain, such as in the subject’s hair, as well as causing a ‘flat’ appearance by softening some facial shadows.

As well, the researchers note that their complete system looks for two types of shadows — facial and foreign — and that it may confuse the two at times. If facial shadows on the subject are ‘sufficiently harsh,’ the system may detect them as foreign shadows and remove (rather than soften) them.

Talking about this issue, the researchers explain:

This suggests that our model may benefit from a unified approach for both kinds of shadows, though this approach is somewhat at odds with the constraints provided by image formation and our datasets: a unified learning approach would require a unified source of training data, and it is not clear how existing light stage scans or in-the-wild photographs could be used to construct a large, diverse, and photorealistic dataset in which both foreign and facial shadows are present and available as ground-truth.

Regardless, the study highlights yet another potential use for artificial intelligence technologies in the photography industry, paving the way for more capable and realistic editing that takes less time to perform than manual editing. A number of studies over the past few years have highlighted potential uses for AI, including transforming still images into moving animations and, in the most extreme cases, generating entire photo-realistic images.

As for this latest project, the researchers have made their code, evaluation data, test data, supplemental materials and paper available to download through the UC Berkeley website.

Via: Reddit

Articles: Digital Photography Review (dpreview.com)

 
Comments Off on Google and UC Berkeley researchers create AI that can remove shadows from images

Posted in Uncategorized

 

Researchers release free AI-powered Fawkes image privacy tool for ‘cloaking’ faces

13 Aug

Researchers with the University of Chicago’s SAND Lab have detailed the development of a new tool called Fawkes that subtly alters images in a way that makes them unusable for facial recognition. The tool comes amid growing concerns about privacy and an editorial detailing the secret scraping of billions of online images to create facial recognition models.

Put simply, Fawkes is a cloaking tool that modifies images in ways imperceptible to the human eye. The idea is that anyone can download the tool, which has been made publicly available, to first cloak their images before posting them online. The name was inspired by Guy Fawkes, the mask of whom was popularized by the movie V for Vendetta.

The Fawkes algorithm doesn’t prevent a facial recognition algorithm from analyzing a face in a digital image — instead, it teaches the algorithm a ‘highly distorted version’ of what that person’s face looks like without triggering errors; it cannot, the researchers say, be ‘easily detected’ by the machines, either.

By feeding the algorithm these cloaked images, it subtly disrupts the machine’s attempt to learn that person’s face, making it less capable of identifying them when presented with uncloaked imagery. The researchers claim their cloaking algorithm is ‘100% effective’ against top-tier facial recognition models, including Amazon Rekognition and Microsoft Azure Face API.

As well, the team says their disruption algorithm has been ‘proven effective’ in many environments through extensive testing. The use of such technology would be far more subtle and difficult for authorities to prevent compared to more conventional concepts like face painting, IR-equipped glasses, distortion-causing patches or manual manipulation of one’s own images.

These conspicuous methods are known as ‘evasion attacks,’ whereas Fawkes and similar tools are referred to as ‘poison attacks.’ As the name implies, the method ‘poisons’ the data itself so that it ‘attacks’ deep learning models that attempt to utilize it, causing more widespread disruption to the overall model.

The researchers note that Fawkes is more sophisticated than a mere label attack, saying the goal of their utility is ‘to mislead rather than frustrate.’ Whereas a simple corruption of data in an image could make it possible for companies to detect and remove the images from their training model, the cloaked images imperceptibly ‘poison’ the model in a way that can’t be easily detected or removed.

As a result, the facial recognition model loses accuracy fairly quickly and its ability to detect that person in other images and real-time observation drops to a low level.

Yes, that’s McDreamy.

How does Fawkes achieve this? The researchers explain:

‘DNN models are trained to identify and extract (often hidden) features in input data and use them to perform classification. Yet their ability to identify features is easily disrupted by data poisoning attacks during model training, where small perturbations on training data with a particular label can shift the model’s view of what features uniquely identify …

But how do we determine what perturbations (we call them “cloaks”) to apply to [fictional example] Alice’s photos? An effective cloak would teach a face recognition model to associate Alice with erroneous features that are quite different from real features defining Alice. Intuitively, the more dissimilar or distinct these erroneous features are from the real Alice, the less likely the model will be able to recognize the real Alice.’

The goal is to discourage companies from scraping digital images from the Internet without permission and using them to create facial recognition models for unaware people, a huge privacy issue that has resulted in calls for stronger regulations, among other things. The researchers point specifically to the aforementioned NYT article, which details the work of a company called Clearview.ai.

According to the report, Clearview has scraped more than three billion images from a variety of online sources, including everything from financial app Venmo to obvious platforms like Facebook and less obvious ones like YouTube. The images are used to create facial recognition models for millions of people who are unaware of their inclusion in the system. The system is then sold to government agencies who can use it to identify people in videos and images.

Many experts have criticized Clearview.ai for its impact on privacy and apparent facilitation of a future in which the average person can be readily identified by anyone with the means to pay for access. Quite obviously, such tools could be used by oppressive governments to identify and target specific individuals, as well as more insidious uses like the constant surveillance of a population.

By using a method like Fawkes, individuals who possess only basic tech skills are given the ability to ‘poison’ the unauthorized facial recognition models trained specifically to recognize them. The researchers note that there are limitations to such technologies, however, making it tricky to sufficiently poison these systems.

One of these images has been cloaked using the Fawkes tool.

For example, the person may be able to cloak images they share of themselves online, but they may find it difficult to control images of themselves posted by others. Images posted by known associates like friends may make it possible for these companies to train their models, though it’s unclear whether there exists the ability to quickly located people in third-party images (for training purposes) in an automated fashion and at a mass scale.

Any entity that is able to gather enough images of the target could train a model sufficiently enough that a minority of cloaked images fed into it may be unable to substantially lower its accuracy. Individuals can attempt to mitigate this by sharing more cloaked images of themselves in identifiable ways and by taking other steps to reduce one’s uncloaked presence online, such as removing name tags from images, using ‘right to be forgotten’ laws and simply asking friends and family to refrain from sharing images of one’s self online.

Another limitation is that Fawkes — which has been made available to download for free Linux, macOS and Windows — only works on images. This means it is unable to offer cloaking for videos, which can be downloaded and parsed out into individual still frames. These frames could then be fed into a training model to help it learn to identify that person, something that becomes increasingly possible as consumer-tier camera technology offers widespread access to high-resolution and high-quality video recording capabilities.

Despite this limitation, Fawkes remains an excellent tool for the public, enabling the average person with access to a computer and the ability to click a couple of buttons to take more control over their privacy.

A full PDF of the Fawkes image-cloaking study can be found on the SAND Lab website here.

Articles: Digital Photography Review (dpreview.com)

 
Comments Off on Researchers release free AI-powered Fawkes image privacy tool for ‘cloaking’ faces

Posted in Uncategorized

 

Google researchers use AI to generate 3D models from random Internet images

13 Aug

Researchers with Google Research and the Google Brain deep learning AI team have published a new study detailing Neural Radiance Fields for Unconstrained Photo Collections (NeRF). The system works by taking ‘in the wild’ unconstrained images of a particular location — tourist images of a popular attraction, for example — and using an algorithm to turn them into a dynamic, complex, high-quality 3D model.

The researchers detail their project in a new paper, explaining that their work involves adding ‘extensions’ to neural radiance fields (NeRF) that enable the AI to accurately reconstruct complex structures from unstructured images, meaning ones taken from random angles with different lighting and backgrounds.

This contrasts to NeRF without the extensions, which is only able to accurately model structures from images that were taken in controlled settings. The obvious benefit to this is that 3D models can be created using the huge number of Internet photos that already exist of these structures, transforming those collections into useful datasets.

Different views of the same model constructed from unstructured images.

The Google researchers call their more sophisticated AI ‘NeRF-W,’ one used to create ‘photorealistic, spatially consistent scene representations’ of famous landmarks from images that contain various ‘confounding factors.’ This represents a huge improvement to the AI, making it far more useful compared to a version that requires carefully controlled image collections to work.

Talking about the underlying technology, the study explains how NeRF works, stating:

‘The Neural Radiance Fields (NeRF) approach implicitly models the radiance field and density of a scene within the weights of a neural network. Direct volume rendering is then used to synthesize new views, demonstrating a heretofore unprecedented level of fidelity on a range of challenging scenes.’

There’s one big problem, though, which is that NeRF systems only work well if the scene is captured in controlled settings, as mentioned. Without a set of structured images, the AI’s ability to generate models ‘degrades significantly,’ limiting its usefulness compared to other modeling approaches.

The researchers explain how they build upon this AI and advance it with new capabilities, saying in their study:

The central limitation of NeRF that we address in this work is its assumption that the world is geometrically, materially, and photometrically static — that the density and radiance of the world is constant. NeRF therefore requires that any two photographs taken at the same position and orientation must have identical pixel intensities. This assumption is severely violated in many real-world datasets, such as large-scale internet photo collections of well-known tourist landmarks…

To handle these complex scenarios, we present NeRF-W, an extension of NeRF that relaxes the latter’s strict consistency assumptions.

The process involves multiple steps, including first having NeRF-W model the per-image appearance of different elements in the photos, such as the weather, lighting, exposure level and other variables. The AI ultimately learns ‘a shared appearance representation for the entire photo collection,’ paving the way for the second step.

In the second part, NeRF-W models the overall subject of the images…

‘…as the union of shared and image-dependent elements, thereby enabling the unsupervised decomposition of scene content into static and transient components. This decomposition enables the high-fidelity synthesis of novel views of landmarks without the artifacts otherwise induced by dynamic visual content present in the input imagery.

Our approach models transient elements as a secondary volumetric radiance field combined with a data-dependent uncertainty field, with the latter capturing variable observation noise and further reducing the effect of transient objects on the static scene representation.’

Upon testing their creation, the researchers found that NeRF-W was able to produce high-fidelity models of subjects with multiple detailed viewpoints using ‘in-the-wild’ unstructured images. Despite using more complicated images with many variables, the NeRF-W models surpassed the quality of models generated by the previous top-tier NeRF systems ‘by a large margin across all considered metrics,’ according to researchers.

The potential uses for this technology are numerous, including the ability to generate 3D models of popular destinations for VR and AR applications using existing tourist images. This eliminates the need to create carefully-controlled settings for capturing the images, which can be difficult at popular destinations where people and vehicles are often present.

A PDF containing the full study can be found here; some models can be found on the project’s GitHub, as well.

Articles: Digital Photography Review (dpreview.com)

 
Comments Off on Google researchers use AI to generate 3D models from random Internet images

Posted in Uncategorized

 

Researchers craft tiny wireless camera that can be attached to beetles

17 Jul

Researchers at the University of Washington have developed a tiny camera that can ride aboard a beetle, offering us a distinct and new view of the world. The lightweight wireless camera can stream video to a connected smartphone at 1 to 5 frames per second and can even pivot up to 60 degrees.

The small camera, which has been used in the real world on Pinacante and death-feigning beetles, records black-and-white images and can even be used in very low light. While an impressive achievement, the specs of the camera itself are nothing to write home about. The monochrome camera streams images that are 160 x 120 pixels.

The device communicates with a smartphone via Bluetooth from up to 120 meters away. In addition to viewing footage, researchers can also remotely control the mechanical arm attached to the camera via an electrical charge. When a high voltage is applied, the material used for the arm bends to the desired position. After the voltage is reduced or altogether removed, the arm will relax back to its original position, like how a human can only keep their head turned for a limited amount of time before needing to return to a natural resting position.

Co-lead author Vikram Iyer, a UW doctoral student in electrical and computer engineering. Image credit: Mark Stone/University of Washington

The tiny camera is a huge feat of engineering; the entire camera system, including the mount, arm and necessary electronic components, weighs about 250 milligrams, which is roughly half the weight the subject beetles can carry naturally. Fortunately, the camera has been designed in such a way that it doesn’t limit the mobility of the beetles or harm them.

At this point, you may be wondering what scientific applications there are for attaching a camera to a beetle. The first-person view from a beetle has allowed researchers to better understand how the subject responds to various stimuli and how it uses vision to approach its environment.

Further, by leveraging its incredibly small camera system, the team also built the world’s smallest terrestrial power-autonomous robot with wireless vision. The robot is ‘insect-sized’ and uses vibrations to move. The team hopes that future versions of the autonomous robot camera could be made without a battery or be solar-powered.

The world’s smallest terrestrial power-autonomous robot with wireless vision. Image credit: Mark Stone/University of Washington

Unsurprisingly, with a such small device, finding a way to power it proved challenging. Researchers turned to the world of insects for inspiration. Flies, for example, dedicate 10 to 20 percent of their total resting energy to power their brains, which are primarily busy with visual processing. In order to efficiently use its limited energy, a fly has a small portion of their overall vision area which sees with high fidelity. In order to see different areas with good detail, a fly must move its head.

This is where the researchers got the inspiration for a movable arm for their tiny beetle camera. Co-lead author of the study, University of Washington doctoral student in electrical and computer engineering, Vikram Iyer, said the following: ‘One advantage to being able to move the camera is that you can get a wide-angle view of what’s happening without consuming a huge amount of power. We can track a moving object without having to spend the energy to move a whole robot.’ To further conserve energy of the system, the camera system includes an accelerometer, which allows the camera to only record images when the beetle moves. In the end, battery life is between 6 and 10 hours.

For more information, members of the American Association for the Advancement of Science can view the full article in the latest volume of Science Robotics. If you’re interested in other electronics-equipped insects, researchers at the University of Washington attached sensors to bees in 2018.

Articles: Digital Photography Review (dpreview.com)

 
Comments Off on Researchers craft tiny wireless camera that can be attached to beetles

Posted in Uncategorized

 
« Older Entries