Choosing Best Face Tracking Software For Android & iOS (+ Examples)
By Alena Arsenova
May 23, 2019
Face tracking software is a solved problem for the desktop. However, its robust performance on mobile devices with limited CPU, memory, and power, or the so-called "constrained devices" is still a challenge that keeps engineers busy.
Businesses want to get this challenge resolved too since mobile is where most consumers are nowadays.
This post explains the aspects of face tracking performance on mobile and provides tips on choosing the best face tracking software for Android and iOS.
You’ll also find the examples that showcase the differences in face tracking performance on mobile.
Face Recognition vs Face Tracking
Before we deep delve into the aspects of face tracking on mobile, one needs to understand the difference between face detection, face tracking and face recognition.
Non-techies often confuse these terms whereas techies may use them interchangeably. However, the functionality of the solutions built with these technologies and their use cases are totally different.
What is facial recognition in 3 words?
Identifying whose face.
The algorithms capture a digital image of an individual’s face and compare it to images in a database of stored records to identify whose face it is.
It’s deployed in phones and payment solutions or for biometric surveillance, in banks, retail stores, airports and other public places to verify a person by an image or detect criminals.
The most common use cases for face recognition:
Surveillance and monitoring
Pass entry systems
What is face tracking in 3 words?
Tracking face location.
Whereas facial recognition is the subject to security and networking solutions mainly, face tracking finds its usage in a variety of apps not aimed to care who you are.
Rather, the technology creates an emphatic camera that “understands” the human face and allows for different face modifications.
The most common use cases for face detection:
3d Avatar generation that visually matches with real user faces and copies their mimics in real time
3D emojis that can be controlled with user facial expressions
Augmented commerce (virtual try ons)
Morphing (solutions to align two different faces/objects to produce an in-between image)
Beautification (makeup try ons, digital camera enhancement)
Analytics (insights into user behaviour, mood or physical parameters like gender, age, ethnicity or even skin condition)
To put it simply:
Face recognition — verifying a person by an image.
Face detection — finding a face in an image.
Face tracking — locating facial features in an image in real time.
All facial recognition SDKs use face detection and tracking.
Not all face tracking SDKs have a facial recognition feature.
In this post, the terms face detection and face recognition are used interchangeably meaning the camera can find a human face and track its movements in real-time.
Aspects of face tracking on mobile vs desktop
Mobile devices have to handle a range of factors that influence face tracking. Among such:
poor resolution capabilities of the mobile cameras compared to the security ones
limited computing capabilities of mobile devices
battery drain and limited memory
extensive range of mobile phones with different camera’s Field Of View for each platform, especially Android
Besides, you need to take into account the short-term operation mode typical for most mobile apps. Users open the app, have it working for a couple of minutes and then close.
With the long-term mode, the operating system can forcibly reduce the system speed to save the battery drain and protect the device from overheating. In the same way, dynamic frequency scaling or CPU throttling is commonly used in computers to automatically slow down the system when possible to use less energy and save battery.
However, real-time video processing on mobile requires significant computing power. If the system reduces the clock frequency, the user experience suffers too.
Non-stop technology optimization.
Engineers are trying to find the golden mean between what’s available and what’s needed both for algorithms and devices to optimize the face tracking performance specifically on mobile.
From getting neural networks trained to detect a face in different conditions to tweaking the technology’s “under-the-hood”, face tracking optimization for mobile should be considered as a range of measures rather than a one-time endeavour.
Parameters to evaluate mobile face tracking software
The technical specification of face tracking SDKs often features such parameters as landmarks, FPS, angles, device and platform support to name but a few.
How does each criterion influence the actual user experience? What does it mean in the context of mobile apps? Here’s the explanation.
Facial landmarks tracking
By detecting a face in the camera, the algorithms extract landmarks or its key facial features such as eyes, eyebrows, nose, mouth, chin, etc. The more landmarks are detected, the more accurate face modification functionality can be applied.
However, tracking a large number of landmarks increases the app’s power consumption and size.
Most often, face detection SDKs support min 68 landmarks. In the context of mobile, 86 landmarks are enough to cover all important facial features and enable robust face tracking.
Banuba’s innovative no-landmark face tracking
Our face tracking technology is not built on landmarks’ tracking. Instead of landmarks, we recognize 37 face characteristics represented as morphs for the default face mesh.
Banuba's face mesh examples
These morphs include 30 parameters with facial expressions and anthropometry plus 7 parameters that describe the position of the face in a photo (head tilts and face angles).
What benefits does it provide?
Since we don’t track landmarks but recognize only 37 face parameters, we achieve a much faster technology performance and better accuracy.
The algorithms need to process only 37 numbers saving the capacity resources.
This saved capacity allowed for our patented anti-jitter technology being used to reduce jitter and achieve stable face tracking specifically on mobile.
We run algorithms several times within one frame which shows the constant data (face itself) and the variable one (the noise). We detect and process the noises within one frame due to which the technology provides no lag.
Low-end device support
People are now holding to their old phones more often compared to previous years. For that reason, in the context of mobile, it’s important to take into account the performance of face tracking on low-end devices.
As of 2019, iPhone 6, iPhone 7, and Samsung Galaxy J5 are the most popular mobile phones in UK, USA and a range of European countries. Xiaomi Redmi Note 4 and Samsung Galaxy J2 Prime keep the leading positions in India and Argentina respectively.
Most face tracking SDKs run perfectly fine on iPhone 8 yet fail to work well on iPhone 5s or Android Nexus 6p. Or they don’t support them at all.
It significantly limits the device reach hence your target audience.
Our face tracking supports iPhone 5S and higher, which covers 95% of all iOS devices available on the market. Banuba's face tracking software for Android supports Android 5.0 or higher with Camera 2 API and Open GL ES 3.0, which covers 75% of all Android devices.
FPS (frame per second)
When talking about face tracking, one should understand the difference between FPS of the camera and FPS of the FRX (face recognition).
FPS of the camera — the number of frames that a camera can stream or capture in a given second.
FPS of FRX — the number of frames that the technology can process and deliver to GPU.
Case in point.
The camera is able to deliver 30 FPS, but the FRX algorithms can deliver only 15 FPS. In the demo test, the camera may show 30 FPS, however, the user experience will be lagging as the technology won't keep up to process the camera's frame flow.
Besides, when it comes to mobile, most vendors forcibly lower the processor frequency under monotonous loads (as face tracking) which leads to the loss of FPS.
The system understands that it's lagging and responds with the frequency increase. One second it gives 30 FPS, the next one — 15 FPS what results in wavelike technology efficiency.
For quality face tracking, you need to ensure the stable FPS number in the course of time.
A low-end mobile device delivers an invariably lower number of FPS because of its limited computing capabilities. Stable 30 FPS is a good mark that provides lag-free face tracking on low-end devices. On high-end ones, it’ll be significantly higher.
Example: Banuba’s mask is fast and stable on iPhone 5s while the other mask trembles and lags.
Banuba | Other
If the user inclines or throws back his head, face tracking may be lost. So does the functionality of the app built on top of it.
If face tracking software supports wide angles, it can detect and modify a face even if the user is not looking straight in the camera. Wide angles provide a better user experience too. You can move, check your angles, and the mask won't fall off. It gives you more freedom and comfort with the app.
The most powerful SDKs support ±90 degrees and full phone rotation.
Example: Banuba mask is lost at 80 degrees vs 70 degrees on the other one
Banuba | Other
Another parameter to check is how face tracking performs on people with different skin colors, i.e. with Afro-American and Asian skin type. The internet is brimming with examples of the technology blamed for racism.
But the answer here is the datasets based on which the algorithms were trained to recognize and modify faces.
We use the balanced datasets, those that include people's faces with different anthropometry (ages, genders, nationalities) to achieve quality performance of face modification technologies, e.g. face masks.
Example: Banuba’s mask is faster compared to the other one.
Banuba | Other
Unless trained to detect faces at low light, the algorithms fail to identify the proper face boundaries if lighting decreases and mix up the face with the background.
There are several ways to approach this either with the camera light switch on — the technology lights the face to ensure face tracking performance. Or it can set ‘low lighting detection” mode so that algorithms wouldn't waste phone's power but rather waited for the proper lighting.
However, the best way is to have the algorithms trained to detect faces in low light. With this, the phone can capture and track the face even if the human eye can hardly find it.
If the user covers the face with a scarf, glasses, hand or other occlusions, face tracking might be lost. However, most SDKs support occlusion to some extent.
Example: Banuba’s mask is accurate with glasses-occlusion while the other one is not.
Banuba | Other
In desktop or outdoor video surveillance systems, the algorithms can track many faces at a time without the loss of efficiency.
On mobile the most optimal number of faces tracked simultaneously without compromising on the quality of performance is 3-4 max.
The most significant criteria for face tracking for Android and iOS devices are as follows:
Stable FPS or lag-free performance
Wide angles supported
Skin color and occlusion
The other important criteria when choosing the best face tracking software include the battery drain, SDK size, and device coverage.
No matter how fast and quality face tracking is, the users won't like it if the app will take too much space on their phone or drain their battery fast.
We'll cover these aspects in the next articles. Stay tuned.