How An Ordinary Camera Phone Can Photograph Objects Hidden Behind Other Things

Physicists have found a clever way to process images from ordinary smartphones that reveals objects hidden behind materials that scatter light


The world of imaging is currently undergoing a quiet revolution. In the last few years, physicists have worked out how to take high resolution images of ordinary scenes using just a single pixel. Yep, that’s just one pixel. Indeed, the same single-pixel technique can produce 3D images and even 3D movies.

If that sounds extraordinary, it’s just the start. This technique requires no lens. Instead, the pixel sits behind a flat sheet of frosted glass that randomises the light from the scene. In a sense, the frosted glass takes the place of a lens.

What is needed, though, is lots of images, which must be processed using a powerful computer to reproduce the original scene.

Now physicists have gone even further. Instead of a single pixel taking lots of images, they show how to do the same thing with lots of pixels taking a single image from behind a piece of frosted glass.

That’s just how ordinary digital camera work. So what these guys have done is show how an image taken with an ordinary smartphone camera can reveal objects that are hidden behind a material that scatters light like frosted glass.

To understand how this works, first some background details about single-pixel imaging. It’s easy to imagine that a single-pixel image taken from behind a piece of frosted glass captures an entirely random pattern of light.

But in fact, a sequence of images taken this way are correlated because they all record the same scene, in a slightly different way.

So the trick to reconstructing the image is to crunch the data from lots of images, looking for the correlation between them. When you find this correlation, it’s a straightforward task to extract an image from it.

And that’s it. One pixel, one piece of frosted glass and one powerful computer. Plus lots of images. Indeed, the more images that are taken, the higher the resolution of the final image.

Now Ori Katz and a few pals at the Institut Langevin in Paris have worked out how to do the same thing with a single picture taken by an ordinary digital camera.

The trick here is to treat the data from each pixel as a separate image. The task then is to look for the correlation between each of these images, just as in the single-pixel imaging techniques.

Of course, in a digital camera, each pixel records the image from a slightly different angle. But even so, they all record the same scene and so produce images that are correlated.

And since the light doesn’t need to be focused, no lens is needed. “The scattering medium effectively serves as the lens,” they say.

That leads to a significant advantage over the earlier technique. Because of the large number of pixels in modern cameras, the imaging process becomes parallel. In fact, Katz and co say the technique parallelises the data acquisition process over a million-fold.

So the new technique consists of taking a single picture of a scene hidden behind some kind of light diffusing barrier and then crunching the data to reveal the original scene.

Katz and co have tested their technique using the camera on a Nokia Lumina 1020 smartphone which has a 41 megapixel sensor. “In these experiments we placed various objects 20cm from the diffuser and simply took an image of the scattered light with the cameraphone placed on the other side of the diffuser,” they say.

There are one or two challenges of course. The lenses on smartphones—not the best in the world to start with—are unnecessary. To reduce their effect, Katz and co place a pinhole aperture in front of the camera lens.

Smartphones also process the data from the pixels, producing a compressed jpeg image. But the team’s technique works even after this kind of processing.

Finally, the correlation between single-pixel images depends on the colour of the light used. So to keep the data processing manageable, the technique requires light of a specific colour. That’s relatively easy to achieve using filters or special light sources.

The results are impressive. Katz and co use their Nokia to produce images of several objects hidden behind light scattering layers, such as frosted glass, onion skin and even chicken breast tissue.

And they even produce images using reflected light (as opposed to transmitting light). To prove this, these guys recorded the light from an object that was scattered off a wall covered in white paint.

Sure enough, the resulting image revealed the object, even though it was essentially around a corner from the camera.

The applications are many. It’s no coincidence that these guys imaged an object behind breast tissue (albeit from a chicken). Medical imaging must be a key area of application.

It should also be useful in situations where lenses are hard to use. The most expensive and problematic part of the Hubble Space Telescope, for example, was its lens. It’s not beyond the realms of possibility that future orbiting observatories—both those that point into space and those that point towards Earth—might have the equivalent of a flat sheet of frosted glass instead.

That should make them smaller, lighter and considerably cheaper. All qualities that are highly sought after by researchers, commercial satellite operators and, of course, governments.

Clearly, a revolution in the offing.

Ref: arxiv.org/abs/1403.3316 : Non-Invasive Real-Time Imaging Through Scattering Layers And Around Corners Via Speckle Correlations


Follow the Physics arXiv Blog by hitting the Follow button below, on Twitter at @arxivblog and now also on Facebook

Next Story — Algorithm Clones Facial Expressions…And Pastes Them Onto Other Faces
Currently Reading - Algorithm Clones Facial Expressions…And Pastes Them Onto Other Faces

Algorithm Clones Facial Expressions…And Pastes Them Onto Other Faces

Want to paste your expression onto a picture of your boss? Your wait is over…


One of the great barriers to remote communication is in conveying facial expressions. Imagine a realistic avatar that represents you in video calls, online chats and web conferences. This character appears in all these environments, adding a realistic presence to your voice.

But here’s the thing: there is simply no way of reading your expression and then recreating it accurately on an avatar or on another face. Animators have the same problem in recreating actors’ expressions on cartoon faces. Consequently, expressions in animated films have to be manually controlled and ordinary video calls are doomed to a dim future of badly lit faces in cluttered bedrooms or boardrooms.

If only somebody could solve the problem of expression cloning…

Enter Yihao Zhang at Shanghai Jiao Tong University in China and a few pals who have created an algorithm that clones facial expressions and pastes them onto other faces. The work raises the prospect of accurately reproducing facial movements and expressions on avatars, cartoon characters and more or less any face.

This is not the first time that computer scientists have attempted to paste an expression from one face to another. But previous efforts have never been entirely successful.

The problem arises because these algorithms measure the way a face distorts when it changes from a neutral expression to the one of interest. They then attempt to reproduce the same distortion on another face.

That can work if the two faces have similar features. But when the faces differ in structure, as most do, this kind of global distortion looks unnatural.

Take, for example, a person with a small mouth and a big smile. This requires a significant distortion which may be inappropriate for many other faces. “If we apply this smile expression to a target person whose mouth is already large in the neutral face, the global warping method may lead to an unnatural “super large” mouth,” say Yihao and co.

What’s more, a change in expression uses a different set of muscles to move the face and this alters the topology of the skin surface, creating dimples, wrinkles and so on. These change the balance of light and shadow on the face and this cannot be reproduced by an algorithm that simply distorts the overall shape of the face.

Yihao and co take a different approach using two different techniques. The first involves the kind of warping already tried but on a local scale rather than an entirely global one.

The algorithm divides the face into regions such as the eyes, mouth, nose etc and measures the distortion in each of these areas independently. In particular, it measures any change in the height-to-width ratio of each facial feature.

It then applies the same warping to each part of the target face. And to ensure that this warping is realistic, it constrains the changes using the measured height-to-width ratio. That ensures that the facial feature never becomes too large or small in relation to the rest of the face.

The second technique produces accurate shadows on the distorted face. To do this, Yihao and co have created a computer model of the facial muscle groups involved in different expressions and the way these change the topology of the face. The algorithm works out which groups have been used to create a certain expression and then adjusts the shadows in the distorted face accordingly.

The results are impressive. Yihaos and co show how their new algorithm can clone expressions and paste them accurately on to other faces creating and compare it to existing methods. “Experimental results show that our proposed algorithm outperforms the existing methods,” they say.

That raises the prospect of a new generation of online communication in which avatars can accurately represent your facial expressions as you speak. It should also be a boon for animators wanting an easier way to make cartoon characters more like the actors who voice them.

It could also be entertaining. Yihao and co use their algorithm to paste expressions on to other people and even to a face in an oil painting (top image). There’s surely fun to be had pasting expressions onto pictures of your friends, significant other, boss etc. Those long winter evenings will just fly by.

Ref: arxiv.org/abs/1503.00088 : Facial Expression Cloning with Elastic and Muscle Models


Follow the Physics arXiv Blog on Twitter at @arxivblog and on Facebook

Next Story — The Origin of Life And The Hidden Role of Quantum Criticality
Currently Reading - The Origin of Life And The Hidden Role of Quantum Criticality

The Origin of Life And The Hidden Role of Quantum Criticality

Quantum criticality must have played a crucial role in the origin of life say researchers who have found its hidden signature in a wide range of important biomolecules


One of the great puzzles of biology is how the molecular machinery of life is so finely coordinated. Even the simplest cells are complex three dimensional biochemical factories in which a dazzling array of machines fill the shop floor.

These machines pump, push, copy, and compute in a dance of extraordinarily detailed complexity. Indeed, it is hard to imagine how the ordinary processes of conduction and electron transport allow this complexity to emerge given the losses that inevitably arise, even in much simpler circuits.

Today, Stuart Kauffmann at the University of Calgary in Canada and a few pals provide some extraordinary new insight into how all this might happen. These show that most biomolecules are quantum critical conductors; their electronic properties are precisely tuned to the transition point between a metal and an insulator.

In other words, biomolecules belong to an entirely new class of conductor that is not bound by the ordinary rules of electron transport, a discovery that has profound implications for our understanding of the nature of life and its origin.

Quantum criticality describes the behaviour of electrons in large molecules when they occupy the exotic state that sits at the knife edge between conduction and insulation. When these molecules are conductors, some of their bound electrons are able to move freely under the influence of an electric field. By contrast, when these molecules are insulators, the electrons are not free to move.

The quantum critical state occurs when the electronic states are balanced between conduction and insulation. In that case, the current is unpredictable and flows in avalanches that can vary in size by many orders of magnitude. It is a condition utterly unlike anything that exists in conventional electronic circuits.

The state of the electrons in a large molecule is determined by the pattern of energy levels the electrons can occupy. Molecules that are conductors have different patterns from those that are insulators. Indeed, it is possible to work out whether a molecule is a conductor or an insulator by calculating the pattern of electron levels.

That’s a difficult task given the number of electrons at work in large molecules. For instance, an ordinary biomolecule such as the protein myoglobin consists of 153 amino acids and over 1000 atoms. However, it has recently become possible to model the pattern of energy levels on powerful computers using NMR data about the physical structure of the molecule and this reveals the tell tale signature of a conductor or an insulator.

The central idea in Kauffmann and co’s paper is that molecules in the quantum critical also have a unique pattern of energy levels that can be determined by this modelling process. The question they set out to answer is: how common is quantum criticality in biomolecules?

The answer is something of a surprise. Kauffmann and co have calculated the pattern of energy levels in biomolecules from a wide range of essential classes and say that the vast majority of them are quantum critical. These molecules include myoglobin, which plays a central role in oxygen storage in the muscles, linoleic acid, sucrose, vitamin D3, vitamin B12, and the largest amino acid leucine and so on.

There are a few molecules that are insulators both by this measure and in experiments. They tend to be structural materials such as silk, a gum-like substance called dextrin and octadecane, an alkane hydrocarbon found in mineral oil. This list also includes DNA, which is otherwise thought of as a wide bandgap semiconductor, or as Kauffmann and co put it, practically an insulator.

The team found only a few biomolecules that are good conductors. These tend to have aromatic rings, such as the hormones testosterone and progesterone.

The team summarise their findings in the following way. “Most of the molecules taking part actively in biochemical processes are tuned exactly to the transition point and are critical conductors,” they say.

That’s a discovery that is as important as it is unexpected. “These findings suggest an entirely new and universal mechanism of conductance in biology very different from the one used in electrical circuits.”

The permutations of possible energy levels of biomolecules is huge so the possibility of finding even one that is in the quantum critical state by accident is mind-bogglingly small and, to all intents and purposes, impossible.

So that fact that most of the molecules they modelled fall into this category is hugely significant. “This shows that chemical and biological evolution selected only a tiny fraction— of the order of 10^−50—of possible small biomolecules and even less for proteins,” say Kauffmann and co. In other words, there is an important evolutionary advantage for molecules in this quantum critical state.

That’s fascinating work that raises many important questions about the nature and origin of life and the role of quantum criticality. For example, what exactly is the advantage that criticality confers? How does this new mode of conduction operate in living things? And why is it such a crucial ingredient in the complex factories of live that evolution has produced?

No doubt Kauffmann and co are pondering these and other questions. They are likely to be joined by cohorts of other chemists, biologists and physicists in the not too distant future.

Ref: arxiv.org/abs/1502.06880 : Quantum Criticality at the Origin of Life


Follow the Physics arXiv Blog on Twitter at @arxivblog and on Facebook

Next Story — Genetic Data Tools Reveal How Pop Music Evolved In The US
Currently Reading - Genetic Data Tools Reveal How Pop Music Evolved In The US

Genetic Data Tools Reveal How Pop Music Evolved In The US

…and show that The Beatles didn’t start the 1964 American music revolution after all


The history of pop music is rich in details, anecdotes, folk lore. And controversy. There is no shortage of debate over questions about the origin and influence of particular bands and musical styles.

But despite the keen interest in the evolution of pop music, there is little to back up most claims in the form of hard analytical evidence.

Today that changes thanks to the work of Matthias Mauch at Queen Mary University of London and a few pals who have used the number crunching techniques developed to understand genomic data to study the evolution of American pop music. These guys say they have found an objective way to categorise musical styles and to measure the way these styles change in popularity over time.

The team started with the complete list of US chart topping songs in the form of the US Billboard Hot 100 from 1960 to 2010. To analyse the music itself, they used 30-second segments of more than 80 per cent of these singles — a total of more than 17,000 songs.

They then analysed each segment for harmonic features such as chord changes and for the quality of timbre, whether guitar or piano or orchestra based, for example. In total, they rated each song in one of 8 different harmonic categories and one of 8 different timbre categories.

Mauch and co assumed that the specific combination of harmonic and timbre qualities determines the genre of music, whether rock, rap, country and so on. However, the standard definitions of music genres also capture non-musical features such as the age and ethnicity of the performers, as in classic rock or Korean pop and so on.

So the team used an algorithmic technique for finding clusters within networks of data to find objective categories of musical genre that depend only on the musical qualities. This technique threw up 13 separate styles of music.

An interesting question is what these styles represent. To find out, the team analysed the tags associated with each song on the Last-FM music discovery service. Using a technique from bioinformatics called enrichment analysis, they searched for tags that were more commonly associated with songs in each music style and then assumed that these gave a sense of the musical genres involved.

For example, they found that style 1 was associated with soul tags, style 2 with hip hop, style 3 with country music and easy listening, style 4 with jazz and blues and so on.

Finally, they plotted the popularity of each style over time.

The results make for fascinating reading. They found that the frequency of style 4 (jazz, blues etc) declined from 1960 onwards. Styles 5 and 13, which relate to rock music, fluctuate throughout this time. And style 2 (rap) is rare before 1980 but expands rapidly after that and becomes the dominant genre for the next 30 years before declining in the late 2000s.

The data allows them to settle some long standing debates among connoisseurs of popular music. One question is whether various practices in the music industry have led to a decline in the cultural variety of new music.

To study this issue, Mauch and co developed several measures of diversity and tracked how they changed over time. “We found that although all four evolve, two — diversity and disparity — show the most striking changes, both declining to a minimum around 1984, but then rebounding and increasing to a maximum in the early 2000s,” they say. Beyond that, their conclusion was clear. “We find no evidence for the progressive homogenisation of music in the charts,” they say.

Instead, they say that the evolution of music between 1960 and 2010 was largely constant but punctuated by periods of rapid change. “We identified three revolutions: a major one around 1991 and two smaller ones around 1964 and 1983,” they say.

The characters of these revolutions were all different with the 1964 revolution being the most complex. This consisted of an increase in popularity of styles 1, 5, 8, 12 and 13 which were enriched at the time for soul and rock-related tags. At the same time, styles 3 and 6 declined, enriched for tags such as doowop.

The 1983 revolution is associated with an increase in popularity of songs with tags such as new wave, disco and hard rock and a decline in soft rock and country tags.

The 1991 revolution is associated with the rise of rap-related tags.

Another question hotly debated by music commentators is how British bands such as the Beatles and The Rolling Stones influenced the American music scene in the early 1960s. Mauch and co are emphatic in their conclusion. “The British did not start the American revolution of 1964,” they say.

The team say the data clearly shows the revolution underway before The Beatles arrived in the States in 1964. However, British bands certainly rode the wave and played an important part in the way the revolution occurred.

That’s fascinating work. Because musicians copy, repeat and modify song styles they like, this leads to a clear pattern of evolution over time. So it should come as no surprise that techniques developed for the analysis of genetic data should work on music data as well. “The selective forces acting upon new songs are at least partly captured by their rise and fall through the ranks of the charts,” they say.

And there is much work to be done. Mauch and co point out that these number crunching techniques are quit general and so could be widely applied to cultural phenomenon, provided the data is available.

For the moment, they have their sights set on further music analysis. Their next task is to gather data going further back in time. “We are interested in extending the temporal range of our sample to at least the 1940s — if only to see whether 1955 was, as many have claimed, the birth date of Rock’n’Roll,” they say.

Worth waiting to find out!

Ref: arxiv.org/abs/1502.05417 : The Evolution of Popular Music: USA 1960–2010


Follow the Physics arXiv Blog on Twitter at @arxivblog and on Facebook

Next Story — Why Hollywood had To Fudge The Relativity-Based Wormhole Scenes in Interstellar
Currently Reading - Why Hollywood had To Fudge The Relativity-Based Wormhole Scenes in Interstellar

Why Hollywood had to Fudge The General Relativity-Based Wormhole Scenes in Interstellar

Interstellar is the only Hollywood movie to use the laws of physics to create film footage of the most extreme regions of the Universe. Now the film’s scientific advisor, Kip Thorne, reveals why they fudged the final footage


Wormholes are tunnel-like structures that link regions of spacetime. In effect, they are shortcuts from one part of the universe to another. Theoretical physicists have studied their properties for decades but despite all this work, nobody quite knows if they can exist in our universe or whether matter could pass through them if they did.

That hasn’t stopped science fiction writers making liberal use of wormholes as a convenient form of transport over otherwise unnavigable distances. And where science fiction writers roam, Hollywood isn’t far behind. Wormholes have played starring roles in films such as Star Trek, Stargate and even Bill & Ted’s Excellent Adventure. But none of these films depict wormholes in the way they might look like in real life.

All that has now changed thanks to the work of film director Christopher Nolan and Kip Thorne, a theoretical physicist at the California Institute of Technology in Pasadena, who collaborated on the science fiction film Interstellar, which was released in 2014.

Nolan wanted the film to be as realistic as possible and so invited Thorne, an expert on black holes and wormholes, to help create the footage. Thorne was intrigued by the possibility of studying wormholes visually, given that they are otherwise entirely theoretical. The result, he thought, could be a useful way of teaching students about general relativity.

So Thorne agreed to collaborate with a special effects team at Double Negative in London to create realistic footage. And today they publish a paper on the arXiv about the collaboration and what they learnt.

Interstellar is an epic tale. It begins with the discovery of a wormhole near Saturn and the decision to send a team of astronauts through it in search of a habitable planet that humans can populate because Earth is dying.

A key visual element of the story is the view through the wormhole of a different galaxy and the opposite view of Saturn. But what would these views look like?

One way to create computer generated images is to trace all the rays of light in a given scene and then determine which rays enter a camera placed at a given spot. But this is hugely inefficient because most of the rays never enter the camera.

A much more efficient method is to allow time to run backwards and trace the trajectories of light rays leaving the camera and travelling back to their source. In that way, the computing power is focused only on light rays that contribute to the final image.

So Thorne derived the various equations from general relativity that would determine the trajectory of the rays through a wormhole and the team at Double Negative created a computer model that simulated this, which they could run backwards. They also experimented with wormholes of different shapes, for example with long thin throats or much shorter ones and so on.

The results provided some fascinating insights into the way a wormhole might appear in the universe. But it also threw up some challenges for the film makers.

One problem was that Nolan chose to use footage of the view through a short wormhole, which produced fascinatingly distorted images of the distant galaxy. However, the footage of travelling through such a wormhole was too short. “The trip was quick and not terribly interesting, visually — not at all what Nolan wanted for his movie,” say Thorne and co.

But the journey through a longer wormhole was like travelling through a tunnel and very similar to things seen in other movies. “None of the clips, for any choice of parameters, had the compelling freshness that Nolan sought,” they admit.

In particular, when travelling through a wormhole, the object at the end becomes larger, scaling up from its centre and growing in size until it fills the frame. That turns out to be hard to process visually. “Because there is no parallax or other relative motion in the frame, to the audience it looks like the camera is zooming into the center of the wormhole,” say Thorne and co.

But camera zoom was utterly unlike the impression the film-makers wanted to portray, which was the sense of travelling through a shortcut from one part of the universe to another. “To foster that understanding, Nolan asked the visual effects team to convey a sense of travel through an exotic environment, one that was thematically linked to the exterior appearance of the wormhole but also incorporated elements of passing landscapes and the sense of a rapidly approaching destination,” they say.

So for the final cut, they asked visual effects artists to add some animation that gave this sense of motion. “The end result was a sequence of shots that told a story comprehensible by a general audience while resembling the wormhole’s interior,” they say.

In other words, they had to fudge it. Nevertheless, the remarkable attention to detail is a testament to the scientific commitment of the director and his team. And Thorne is adamant that the entire process of creating the footage will be an inspiration to students of film-making and of general relativity.

Of course, whether wormholes really do look like any of this is hard to say. The current thinking is that the laws of physics probably forbid the creation of wormholes like the one in Interstellar.

However, there are several ideas that leave open the possibility that wormholes might exist. The first is that wormholes may exist on the quantum scale, so a sufficiently advanced technology could enlarge them in some way.

The second is that our universe may be embedded in a larger multidimensional cosmos called a brane. That opens the possibility of travelling into other dimensions and then back into our own.

But the possibility that wormholes could exist in these scenarios reflects our ignorance of the physics involved rather any important insight. Nevertheless, there’s no harm in a little speculation!

Ref: arxiv.org/abs/1502.03809 : Visualizing Interstellar’s Wormhole


Follow the Physics arXiv Blog on Twitter at @arxivblog and on Facebook

Sign up to continue reading what matters most to you

Great stories deserve a great audience

Continue reading