As the remote work style is becoming more and more common, people have more opportunities of having online meeting. We want to make the online meeting more interactive and enjoyable, and that is where Livemoji can help you!
What it does
It will detect your face emotion and show the emoji on the meeting website at that person's space with a notification sound, so everyone can see each other's emotions easily and it is simply enjoyable just to try to change your face emotion to produce the emoji.
How we built it
We built on top of the twilio programmable video API project of React (Github). We take screenshot of the current user's webcam and added API request to Cloud Vision API to get the face detection result and show up the emoji depending on the result we get. We also did a machine learning training through AutoML in order to make it able to detect waving hands so we can show that emoji as well.
Challenges we ran into
- Finding a way to see the result of detection. We initially tried to use the Google Video Intelligence, but it turned out that it is so hard to build up the live stream pipeline. That's why we moved to using Vision API with webcam screen shot.
- The training part using the AutoML. We need to supply enough high-quality data which we should have improved on since the result we got was not that satisfying at the end.
Accomplishments that we're proud of
- First of all, the idea itself that we came up with because it is a lot of fun for many people.
- Integrating Twilio video meeting with Cloud Vision API to judge on the user's face emotion.
- Being able to use AutoML to actually training a model and learned how easy the interface is to use.
What we learned
- How to use Twilio to integrate video meeting interface
- How to use Vision API and its output to get some insights from pictures
- How to use AutoML to train a new model that is in demand
What's next for Livemoji
- We are thinking of raising hands to show some special effects so it is easy for the professor to see which can be used at universities. This way, even though you are taking the course virtually, you will still have the classroom experience of actually raising your hands to ask questions.
- We can add a game like feature into this meeting so people can enjoy killing the time before all the members got there.
Try It out
automl, firestore, node.js, react, twilio, vision