- Mariam Hakobyan
- Sergei Volodin
- Arash Atrsaei
- Salil Apte
- Frederike Dümbgen
- Paolo Prandoni
- Mihailo Kolundzija
We introduce the Hand Angel project with the main purpose of developing and keeping good hand hygiene habits. This epidemic introduced a lot of negative points to our life recently, but we should also accept that from some aspects it has reminded us how to act in life, how properly wash hands, cover our face during coughing, so that others are not affected and of course take care of family and elderly people and stand up to take our portion of the responsibility of everything next to us. But, unfortunately, good habits do not tend to last long, and people tend to forget things if they don't really experience them. It is interesting how people will conduct their hand-washing habits after many years. That is why we put this effort to think of ways in which healthy habits can be developed and retained, even after the epidemic ends.
What it does
With the help of a phone app and a smart wearable device worn on the wrist, IMU accelerometer signals are used as the main data source to detect two types of hand movements, touching the face and washing hands. Machine learning approach is used to design a classifier, which recognizes these two events with an accuracy of 97% and 96% respectively. Each time when the user touches his or her face, he or she will receive a vibrating notification from the wearable. Such an alert will help the user to train their subconscious minds in recognizing the face touch action and eventually train themselves to do it less.
After several instances of detecting face touch events, the system generates a reminder to wash the hands. The app will also generate a reminder when the phone is standby for a given amount of time, assuming that the user’s hands get dirty with time, regardless of touching the face. Our system will detect if the user followed the reminder by tracking the hand movements after the notification to be sure to remind a bit later or praise the user. The user will get a positive score for every successful hand wash, with the aim of improving the user compliance.
The notification system also uses the information of geolocation matching; every time the user enters a designated place (a place where he or she spends most of their time in a day) the system generates a reminder for the user to wash hands. Assuming the designated places to be clear and safer, the app also adjusts the threshold of the face touching occurrences in reminder generation. This threshold is lower in designated places and higher when the user is outside them.
Based on the positive rewards received from the successful hand washing and negative scores from the number of hand touches, the app also generates statistical reports about hand hygiene over time, thus providing crucial feedback during the learning process. The system generates charts about the users hand hygiene for each day and gives a score for each day's performance. The system will enable the users to create a community of friends and share the daily improvements and compete with scores of hand hygiene. By introducing a community aspect, our app will create a feeling of positivity and friendly competition around a routine habit of hand washing, thus making it relevant beyond the current pandemic.
How we built it
We started by creating roadmaps for how we should design the three main components of the app, namely, the design of classifiers and when to trigger them, how to generate a list of designated places and check for them using geolocation, and the design of the user experience in terms of feedback and gamification. After formulating these roadmaps as flowcharts, we proceeded to the data collection phase.
Three datasets were generated with a wearable IMU device to train the classifiers, consisting of hand-wash and face-touch instances and other activities such as walking, running, writing with pen, typing, etc. Of the three datasets, the first two measurements were for washing hands and the last one for face touching. Flowcharts for the roadmaps and the data collection protocol can be accessed in the github repository.
To train the classifiers, the accelerometer signal was first filtered and preprocessed, and 15 time domain and 2 frequency domain features were extracted from it. Finally, two support vector machine classifiers, based on quadratic kernel function, were trained on these datasets, one each for detecting hand washing and face touching respectively. Finally, the accuracy of these classifiers was tested on these datasets using a 5-fold cross validation.
We used Fluid UI to design the user interface prototypes for our app, with the icons being created using Adobe Illustrator. The interface and the notification system is based on the idea of using positive reinforcement learning for inducing and maintaining good habits. Some apps are more addictive, because they seem more "real" and feel more "important" than others. This is because of an experience of social presence and connection via social networks and reward systems. Nowadays, it is hard to spend our time on apps that are too "boring", and often habit trackers and productivity systems are forgotten and abandoned. Therefore, we must make our app fun in order to fight COVID-19. We replace "addiction" daemon into an "angel", an entity helping you to develop better habits and not forget them, because you don't want to. Our technology is based on the same phenomenon as the addiction caused by social networks, but we use it to do good.
Challenges we ran into
The first major challenge for inducing a new habit and thus for us, is that of user compliance. We tried to overcome this with a feedback-based system, designed around a community of users, thus introducing a social and positive aspect in people’s routine habits. Considering the short duration of the competition, another major challenge for us was to agree upon necessary and nice-to-have features for the application quickly, so that we could go ahead with the development. From the algorithmic standpoint, having the watch on a non-dominant hand can be a challenge, since the app is likely to underestimate the number of hand touches in such a case.
Accomplishments that we're proud of
We successfully identified a relevant problem and managed to create and follow a structured roadmap to solve it. We collected data from hand movements and developed classification algorithms which provide a high accuracy. Finally, we could develop prototype GUI to showcase the app features and functions. All of this within 72 hours!
What we learned
During this hackathon, we have learned the how to deal with the following topics: Defining a problem based on the people’s needs and what is missing in the society to solve them Brainstorming to find the solutions Narrowing down the solutions to the ones that are more feasible and cost-effective Breaking down the problem and the solution into several blocks Dividing the blocks between the team members based on their skills Making a prototype that simplifies the whole process but at the same time represents our objectives and obstacles Team working skills
What's next for Hand Angel
Our next step is to translate our algorithms into a real-time online-learning process. Following this, we will improve and develop further the gamification aspect of our app.
Try It out
android, fluid-ui, matlab, python, tensorflow