Recently, ToF cameras have become an integral part of modern flagship smartphones. But how do they work, why are they needed, and where can they be used?
“They must be good for something, right?” – most users will shrug. The fact is that very little information is published about ToF sensors. Only fellow journalists or experts in the know can clearly explain the need for them. Let’s bring some clarity to this issue. Perhaps part of the text will contain technical terms, but I will try to explain as simply as possible what the ToF camera is, where you can apply its capabilities and how to see it in action. Let’s go.
- This article is published with the support of Konica Minolta – the world’s No. 1 manufacturer of production printers.
What is ToF?
Remember submarine films? In them, people usually had some kind of sonar equipment used to detect enemy ships. The device sends a sound “ping” into the water and waits for the signal to reflect. Depending on the time it took for the sound wave to return, the specialist tried to determine the location of the enemy submarine.
Just like a sonar, a ToF camera measures the time it takes for light to bounce off an object. The camera then uses the delay data to determine the distance. Thus, the ToF sensor creates a three-dimensional map of the world.
Therefore, ToF cameras are also sometimes referred to as the “depth camera”. A ToF camera (Time of Flight) is a sensor that can emit some kind of light in the infrared spectrum and then record the speed of its reflection from the object. The sensor itself consists of two parts: the first is a diode emitting infrared light, and the second is a special photosensitive matrix.
The camera measures the time after which the light reflected from the object returns to it. Based on calculations accurate to the nanosecond, the device can not only accurately estimate the distance of individual objects from the sensor, but also determine their shape. True, it is very similar to sonar. Only in this case we launch light to the object instead of sound.
Sensors operating on a similar principle are also used in self-driving cars or even space probes built by NASA. Miniaturization efforts and relatively low manufacturing costs mean that this technology may appear in your pocket today. The fact is that ToF sensors are already in many top-end smartphones. Here are some mobile devices equipped with ToF cameras:
- Samsung Galaxy S10 5G
- Samsung Galaxy Note10+
- Samsung Galaxy S20+
- Samsung Galaxy S20 Ultra 5G
- LG G8 Thin Q
- OPPO RX17 Pro
- Honor View 20
- Huawei P30 Pro
- Huawei Mate 30 Pro
- Huawei P40 Pro
- ОPPO Reno
- Sony Xperia XZ4
I am sure that every year this list will only expand, because the potential of this sensor is huge. Perhaps soon Google will also install such a camera in its Pixel. There is also hope that Apple will pay attention to the ToF sensor and contribute to popularization of this technology, as well as make full use of its capabilities. No wonder that a similar sensor appeared in the new iPad Pro.
Well, the theory is clear, let’s now take a closer look at what the ToF camera is capable of in practice. You need to understand that you can’t just turn on this sensor, but you can still use it in some applications, even when shooting with a camera.
How can I use a ToF camera?
Measure the distance
On some Samsung (including Note10+, S20+ and S20 Ultra) and Huawei smartphones (including P30 Pro, Mate 30 Pro, Huawei P40 Pro), you can install a distance measurement application called Measure.
Everything works quite simply and with impressive accuracy. As part of the tests, I measured the length, width and diagonal of the screen of my TV using this application and Samsung Galaxy S20 Ultra showed precisely the right dimensions.
Blur background when taking photo
3D visualization system can be used to create a digital depth of field so that you can add a blurry background to your photos or videos. At the moment, this only works for photos, not videos. The function also allows us to get rid of excess objects or backgrounds that spoil the impression of the picture.
In addition, on some phones the ToF sensor supports autofocus or allows you to more accurately blur the background in portrait mode. In short, it helps you take better selfies.
Motion tracking
LG G8 ThinQ smartphone has a ToF sensor located on the front panel, which is used, among other things, to detect gestures performed in the air. I don’t really want to talk about the possibility of using such gestures, it’s better to see for yourself.
There are also a few motion games on Google Play, similar to those supported by the Microsoft Kinect controller.
Creation of 3D models and augmented reality
You can talk endlessly about the ability to create 3D models using a ToF camera. Already you can use your selfie photos from different angles to create a model of your face and even print it if you have a 3D printer at hand. Of course, someone will say that these are just games. But it is quite realistic that in the future ToF cameras well help us create parts for cars or other mechanisms.
Virtual and augmented reality is knocking more and more persistently on our doors. Games with support for are also becoming more widespread on Android and iOS. In particular, in Play Store you can already find a large number of applications and games that support Google ARCore platform. It is enough to download and install the Google Play Services app for AR, and similar games and applications in augmented reality will be available to you. Just the use of a ToF camera makes it possible to fully enjoy augmented reality.
In turn, in the new Apple iPad Pro, the depth sensor is responsible for more accurately superimposing virtual objects on the camera image in AR mode.
Smartphone as a night vision device?
Currently, for most devices, the ToF sensor works almost imperceptibly for the user, supporting only functions that require determining the depth of the scene.
However, on Google Play I found a great Night Vision application that allows you to use a 3D scanner in an unusual role. I installed it on my Huawei P30 Pro.
The program in real time displays what the ToF sensor captures. Since it uses infrared light that is invisible to the human eye, the depth map is clearly visible even in absolute darkness.
I was surprised at how large an area can be scanned by a tiny sensor. In the blink of an eye you can create a three-dimensional map of the entire room! ToF-camera easily detects the shape of small objects at a distance of 4-5 meters.
Especially such a night vision device will appeal to your child. I am sure that he will immediately run to explore the darkest corners of your apartment or house.
ToF camera as a biometric sensor
The ToF camera is most often implemented in the form of two miniature holes in which an infrared diode and a matrix are hidden. The question arises here: “Why does Apple not want to use this technology instead of its TrueDepth system, which has numerous sensors that occupy more than half the width of the phone screen?”
The reason is probably due to poor scan quality. ToF cameras used in smartphones currently have a very low resolution (up to 240p), resulting in a very pixelated image.
As you can see, the details of such a scan are too small for the software to accurately determine facial features.
But, personally, I do not see why the face scanner could not work in hybrid mode. Theoretically, at the same time, while the selfie camera analyzes facial features and detects open eyes, the ToF sensor can check whether it is dealing with a three-dimensional object of a certain shape and size, and not with a photograph.
Such a scanner is potentially very difficult to fool, and at the same time it is fast, cheap and non-invasive. Therefore, I would not be surprised if in the near future Apple will use ToF technology to create the next-generation Face ID scanner.
Huawei P40 Pro has the exact same sensor, it is located between the dual selfie camera and takes part in the unlocking process. And I was wondering why unlocking with Huawei P40 Pro is so much faster than with P30 Pro… And the reason is ToF. In addition, you can’t deceive the new flagship with the help of a photo. I checked.
Future Applications for Aspiring Moviemakers
Anyone who uses their smartphone to shoot video will be extremely interested in ToF, since 3D technology can be used to add a background depth of field. Especially if the process can be transferred to post-production, as this provides greater flexibility.
Also, ToF features are of potential interest to those who add computer effects to their videos. For example, when tracking motion in a video, it’s best to focus on the subject as accurately as possible. Using a 3D map should allow us to capture everything in focus, and later add the necessary background blur and special effects.
This is just an assumption on my part, but 3D scanning certainly has enormous potential for improving the video shooting process. This has already been proven by modern cinema and I hope it will soon appear in the consumer segment.
The reason that these features are not yet available in smartphone cameras when shooting video is probably due to the need for increased processing power. Testing has proven that when using the ToF sensor, smartphones tend to heat up. And since they do not have active cooling, it can be a problem.
But with the increasing power of mobile processors, I think ToF cameras will become more widely used in photo and video applications.
One thing is certain – ToF has great potential
The fact that the smartphone can become an ultra-precise distance measuring device or a night vision device means that I will never buy a phone again without a ToF sensor. This is one of the most interesting technologies of recent years, and we are only getting started.
It is impressive that such a small sensor can do so much. I cross my fingers so that manufacturers and developers will continue working on improving its capabilities. I am sure that we will hear more than once about the new application of the ToF camera. And who will say now that this is a useless addition to the basic set of smartphone features?