You may have heard the old story that you can hear what was said while painting old paintings. The sound waves would somehow show up in the paint. Sounds plausible, but it’s not true. In this story it’s the other way around: it sounds impossible to extract sound from a photo, but it’s true.
Scientists from four US universities have created a technique they call Side Eye. With this software they extract audio from photos and silent videos. To remove the sound from the image, they use the stabilization of the lens. The audio that Side Eye extracts from the photo is said to be 80 to 99 percent correct. So you can extract the sound of a car from a photo – even if it is a very short fragment.
A photo that you would like to hear | Photo: © TopGear
To make videos and photos sharper, most phones these days have image stabilization. This technique causes the lens to move slightly to keep the image stable. Sound waves also cause the lens to move slightly. By analyzing the movement, scientists can reproduce environmental sounds from during the recordings.
Not only with a video, also with a photo
With a video it is easy to understand how you can determine sound through the vibrations. But what about a photo? Phone cameras use something called rolling shutter. The camera does not take one photo, but actually a lot of small layers underneath each other. By analyzing these layers, the movement of the stabilization can be determined.
You can almost smell this | Photo: © TopGear
It can be quite fun for the car enthusiast; you can extract the annoying TikTok and Instagram sounds under a video and hear the original engine sound. Or you can hear what a car sounds like from a photo. It is also quite creepy: a video that you have sent without sound can suddenly be listened to. Then people suddenly hear that you accidentally called that Lamborghini a Ferrari.
What can you do about it?
If you use a camera without image stabilization, you are safe. Even if your camera does not use a rolling shutter, you cannot distill sound from a photo. Just go through life with a Nokia 3310 from now on, then no one can do anything to you. Do you really need TikTok and Instagram when you have Snake on your phone?
