LokaPulse™️

‍

‍

This page is dedicated to the journey of LokaPulse^TM, from idea to testing to deployment and improvements.

We dive deep into backend and frontend technologies, challenges and eureka moments. For people interested in the big ideas and key takeaways, look to the IDEA and TODAY sections, as well as any highlighted areas.

Idea

Mobile is just scratching the surface of machine learning capabilities. LokaPulse^TM was designed to showcase the possibilities of ML and Flutter by giving users an Apple and Android app that would measure their heart rate in an innovative and beautiful way.

With a simple 15-second video, LokaPulse^TM can extract a user’s BPM information, categorize the data, and compare it to previous videos, giving anyone with a smartphone the ability to see and track their heart rate.

The idea for LokaPulse^TM started when a Loka developer stumbled upon a method called Eulerian Video Magnification. “Super Seeing”, as it’s also known, amplifies and visualizes temporal variations in videos that are difficult or even impossible to see with the naked eyes. These visual variations include motion and color.

As Loka developers and the team behind LokaPulse^TM, we wanted to see for ourselves how this method worked. We implemented the algorithm from scratch and applied it to several different videos. The results were inspiring. Even the most subtle variations were amplified beyond what we imagined.

Though the potential applications for this technology were endless, what really caught our attention was the color magnification. The idea of extracting “invisible” biosignals from a video was exciting to say the least. And we knew it could be used as a heart rate calculator.

This is the moment that sparked the big question, what would happen if everyone with a phone could measure their heart rate with a selfie video? What else could this unlock for people’s daily health - and for life sciences? And so, the idea of developing LokaPulse^TM was born.

Journey

Going from idea to fully-functioning app is never short on challenges. The question is, when will the challenges emerge and how will you respond.

Our first step was to do an in-depth analysis of the state of the art (SOTA) and available datasets. After we explored all kinds of possible bio signals that can be extracted from video, we decided to start with heart rate estimation.

The research into extracting a heart rate signal from video led us to a technology called remote photoplethysmography (rPPG) which relies on the reflection of light on the subjects skin.

This technology is the most prevalent in SOTA and seemed to benefit from the increased variations provided by Eulerian color Magnification. However, as we dove deeper into SOTA methods and comparative studies, we noticed a troubling pattern: The improvements provided by Eulerian Magnification often didn’t outweigh its limitations.

The improvements provided by Eulerian Magnification often didn’t outweigh its limitations.

Here’s what we encountered as we explored SOTA methods and tools:

• Increased variations caused by “noise”, e.g. motion and light conditions.
• Performance seemed highly dependent on parameter tuning reducing the system’s robustness for use in the wild.
• The tools we came across relied on extremely outdated library versions
• The tools had serious bugs that prevented efficient use.
• The tools they simply had very few methods implemented

To overcome these obstacles, we developed a pipeline architecture based on these tools where we could plug and play at different stages to compare performances of the different methods and processing steps. For the comparisons, this pipeline architecture could be used in a benchmarking tool that allowed automated comparisons for different parameters.

Benchmarking different methods and parameters required labelled data with which to compare the predictions. We evaluated commonly used datasets focusing on their similarity to our use case and on subject diversity. We started testing with a dataset that would be our ideal scenario and from there moved to videos captured in real situations.

Through experiments and research, we concluded that SOTA methods tended to require fine tuning for each specific dataset – and most tended to increase the pre and post processing complexity by adding extra stages while relying on the same core principles. Since we required near real time results as we iterated, we leaned toward simpler approaches, even if they were not the most recent and best performing methods.

Here’s the flow of the solution we chose

• Image rescale and crop on the app.
• Use a Deep Learning model to detect and extract user’s face.
• Extract the PPG signal.
• Use Welch’s method to obtain the power spectrum and calculate the heart rate.

As we moved to real world application, we faced three critical obstacles:

• Variations in camera quality between phones (e.g. resolutions and frame rate), which prevent video compression after recording.
• Lighting conditions - the most persistent and stubborn problem.
• And the biggest limitation of the app: Performance deteriorates sharply for darker skin types

‍

The issue of skin tone stems from using rPPG to extract the heart rate. Since rPPG relies on the difference between reflected and absorbed light, skin types that absorb more light provide a signal with lower Signal to Noise Ratio (SNR) damaging performance, similarly to what happens under poor lighting conditions.

This isn’t a new problem. Products like Oxymeters, which became more prevalent during the pandemic, rely on PPG signals and are charged with the same issues. This continues to be a challenge to LokaPulse^TM and other programs using PPG signals will need to face and overcome.

Framework

At the beginning of any software project, there is always a big question: Which mobile framework/language is better for my use case – and is it good enough for the long term? For LokaPulse^TM, we decided that we wanted a multi-platform framework and narrowed it down to two options: Flutter or React Native.

Both are really good options for our use case, offering multi-platform support and large communities. Ultimately Flutter was the right choice based on the following criteria:

Support: thanks to the Flutter canvas approach, we don't need to worry about outdated APIs or integrations with a specific platform. Flutter provides a white canvas and puts us in charge of rendering each pixel on the screen.

Performance: 60fps (or even 120 fps) is a really attractive feature for smooth user experiences.

Custom design: We could create our own designs, so we didn't need to reuse the platform specific UI components.

Time-to-market: Flutter is known for lightning fast development.

For us, building LokaPulse^TM became as much of a way to showcase the capabilities of machine learning as it was to highlight the outstanding performance of Flutter.

Frontend

At its core, LokaPulse^TM is a Flutter app that measures a users heart rate via video recording of their face. The application provides a way to record a video, process the video, and present all the data connected with previous measurements.
To improve the performance and user experience, the front end is capable of “cropping” the video to only send the required information, the white square that contains the face. With this we ensure that we are not using extra data of your carrier plan and also the Face detection and heart rate algorithms run faster than using the original video.

Tagging

LokaPulse^TM uses a custom ffmpeg command for tagging. After recording a video, users are prompted to select a tag, which defines the state that they were in while recording the video.
Tags can be used to track whether a user was stressed, relaxed, exhausted, excited, etc., so users can better understand their results and trends over weeks and months. LokaPulse^TM also gives users the option to manually enter a comment, or even use speech-to-text, for better explanation of the moment when the measurement was taken.

When you’re running, your heart rate will be elevated. LokaPulse^TM allows users to tag these measurements and separate them from categories like ‘chill time’.

Speech-to-text

Speech-to-text offered our team some challenges. The package that we used worked differently on Apple and Google platforms. Little tweaks and workarounds were necessary. For example, LokaPulse^TM would stop working after a few moments of silence on Android phones. So after 30 seconds of silence, we made it so the speech-to-text would stop and re-engage.

Anomalies

We built a way to detect anomalies into LokaPulse^TM. If a measurement result is not in the range of 60 to 120 bpm, the application gives the user a notification that the heart rate is not in their normal range or inaccurate due shaking or poor lighting conditions.

Seeing & Managing Results

Users can find their data in a list organized by date, tag, and heart rate. Pressing on any measurement gives users the option to see, edit and delete past notes.

Visualization

For a better user experience, we created a separate screen where data is presented as a bar chart for the current week and month, with multiple tag selection capabilities. Users can also see their average, maximum and minimum measurements.

Implementation

LokaPulse^TM is coded on Flutter 2.0, which implements null safety. For the state management of the application, we have two providers: The first is used for authentication. It allows the app to respond to a user based on whether they’re logged in or not. This provider is based on the Amazon Cognito. The second is the results provider, which saves the data that is being fetched from the backend and being modified through user interactions.

Logic & Widget

When developing LokaPulse^TM, we had in mind optimization and reusability. With Flutter you can extract all your widgets, functions and screens in a different directory, this will make it easier to import, test and reuse all that code keeping the code clean and separate between business logic, utilities and screens.

Testing

LokaPulse^TM is built to look great and respond intuitively on phones and tablets. So for widget testing, we called the same tests on two different resolutions – one is for a phone and the other on a tablet. Mocking the files was an easy task thanks to the mockito flutter package. This package was essential to our testing. Not only did it help us understand how the app would look on phones and tablets, it also helped us fix rendering issues that might occur on some devices.

We used all of the phones and tablets available to us, including holding weekly internal testing sessions at Loka where volunteers would test on their own devices. This helped us spot and fix problems that came up on a greater number of phones and tablets. Additionally Loka Pulse^TM has Firebase Crashlytics, so if the app crashed on someone’s device or if exceptions were spotted, we could easily track the issue and where it occurred.

UX

Leveraging Flutter capabilities of render at an outstanding 60fps (or 120fps if your device is capable), we decide to include animations, transitions and sounds to enrich que user experience and increase the probabilities of reusability. The hero animation is probably the most engaging, as it animates widgets between screens.

Backend

Serverless for more savings

We chose to implement a serverless architecture for LokaPulse^TM. Serverless architectures offer a number of advantages over traditional cloud-based or server-centric infrastructure, which aligned well with the goals for Loka Pulse^TM.

• Serverless is managed by the cloud provider, making it easier to quickly build a proof of concept project.
• In traditional architectures, the developer has to provision and pay in advance for server capacity needed – regardless of whether it gets used or not.
• With serverless, you only pay for what you use, which helps drive costs down in many applications.
• Serverless scales automatically as the number of users and the usage increases.

Backend flow

• The mobile app gets a signed url that allows it to upload a video to an S3 bucket.
• When a video gets uploaded to the S3 bucket, it triggers the invocation of the Heart Rate Calculation Lambda function, which computes the heart rate for that video and saves the result in a DynamoDB table
• The mobile app can then get the results from this table, and associate tags and notes with each video.
• The authentication of the users is managed by AWS Cognito.
• AWS QuickSight is used to create interactive dashboards that give us powerful BI data, such as number of users, number of uploaded videos and high level statistics about the uploaded videos.
• AWS Backup automatically backups data from DynamoDB

Today

LokaPulse^TM was selected as a showcase application at Flutter Engage 2021. Run by Google, Flutter Engage focuses on the biggest product announcements and most exciting technical demos. LokaPulse^TM is now part of the Google Favorites Flutter Showcase apps, which is a great achievement for Loka and our Flutter team.

All that said, we are just scratching the surface of what’s possible with heart rate calculation, Eulerian Magnification, machine learning, and expressing it all visually with frameworks like Flutter. In terms of biosignals, the next logical steps are working on extracting the heart rate variability, the breathing rate and the “holy grail”, blood pressure.

With all these in place, we could provide users a very complete analysis of their current health condition. However, before moving to extracting more signals, we need to improve the current application.

Our ongoing work is focused on tackling the main sources of noise, like camera stabilization and background extraction, and the limitations of rPPG, including the need for uniform lighting conditions and the degraded performance on users with darker skin colors. Our team is excited to evolve LokaPulse^TM and continue to show how this technology can improve people’s everyday lives.

Reach out to learn more or partner on innovative products. Contact us.

Appendix

An ongoing addendum of applications for LokaPulse^TM technology.

Detect Deepfakes

According to researchers and publications like Forbes and Independent, Deepfakes are the most dangerous crime of the future. The good news is that all these artificially generated videos leave behind some signature on the PPG signal. Using our technology, we could better identify deepfakes, decrease the spread of fake news, and help protect online communities.