The COVID-19 pandemic has caused mass panic and is leaving everyone paranoid. In a time like this, simply leaving the house leads to a high risk of contracting a fatal disease. Survival at home is also not easy: buying groceries is frightening and online ordered necessities take ages to arrive. The current delivery system still requires a ton of human contact and is not 100% virus free. All of these issues are causing a ton of paranoia regarding how people are going to keep their necessity supply stable. We wanted to find a solution that garners both efficiency and safety. Because of this, drones came into the picture(especially since one of our group members already had a drone with a camera). Drone delivery is not only efficient and safe, but also eco friendly and can reduce traffic congestion. Although there are already existing drone delivery companies, current drone navigation systems are neither robust or adaptable due to their heavy dependence on external sensors such as depth or infrared. Because of this, we wanted to create a completely autonomous and robust drone delivery system with image navigation that can easily be used in the market without supervision. In a dire time like now, a project like this could be monumentally applied to bring social wellbeing on a grand scale.
What it does:
Our project contains two parts. The first part is a deep learning algorithm that allows the drone to navigate images taken with a camera which is a novel and robust navigation technique that has never been implemented before. The second portion is actually implementing this algorithm into a delivery system with firebase and a ios ecommerce application.
Using deep learning and computer vision, we were able to train a drone to navigate by itself in crowded city streets. Our model has extremely high accuracy and can safely detect and allow the drone to navigate around any obstacles in the drone’s surroundings. We were also able to create an app that compliments the drone. The drone is integrated into this app through firebase and is the medium in which deliveries are made. The app essentially serves as an ecommerce platform that allows companies to post their different products for sale; meanwhile, customers are able to purchase these products and the experience is similar to that of shopping in actual stores. In addition, the users of the app can track the drone’s gps location of their deliveries.
How I built it:
To implement autonomous flight and allow drones to deliver packages to people swiftly, we took a machine learning approach and created a set of novel math formulas and deep learning models that focused on imitating two key aspects of driving: speed and steering. For our steering model, we first used gaussian blurring, filtering, and kernel-based edge detection techniques to preprocess the images we obtain from the drone's built-in camera. We then coded a CNN-LSTM model to predict both the steering angle of the drone. The model uses a convolutional neural network as a dimensionality reduction algorithm to output a feature vector representative of the camera image, which is then fed into a long short-term memory model. The LSTM model learns time-sensitive data (i.e. video feed) to account for spatial and temporal changes, such as that of cars and walking pedestrians. Due to the nature of predicted angles (i.e. wraparound), our LSTM outputs sine and cosine values, which we use to derive our angle to steer. As for the speed model, since we cannot perform depth perception to find the exact distances obstacles are from our drone with only one camera, we used an object detection algorithm to draw bounding boxes around all possible obstacles in an image. Then, using our novel math formulas, we define a two-dimensional probability map to map each pixel from a bounding box to a probability of collision and use Fubini's theorem to integrate and sum over the boxes. The final output is the probability of collision, which we can robustly predict in a completely unsupervised fashion.
We built the app using an Xcode engine with the language swift. Much of our app is built off of creating a Table View, and customized cell with proper constraints to display an appropriate ordering of listings. A large part of our app was created with the Firebase Database and Storage, which acts as a remote server where we stored our data. The Firebase authentication also allowed us to enable customers and companies to create their own personal accounts. For order tracking in the app, we were able to transfer the drone’s location to the firebase and ultimately display it's coordinates on the app using a python script.
The major challenge we faced is runtime. After compiling and running all our models and scripts, we had a runtime of roughly 120 seconds. Obviously, a runtime this long would not allow our program to be applicable in real life. Before we used the MobileNet CNN in our speed model, we started off with another object detection CNN called YOLOv3. We sourced most of the runtime to YOLOv3’s image labeling method, which sacrificed runtime in order to increase the accuracy of predicting and labeling exactly what an object was. However, this level of accuracy was not needed for our project, for example crashing into a tree or a car results in the same thing: failure. YOLOv3 also required a non-maximal suppression algorithm which ran in O(n^3). After switching to MobileNet and performing many math optimizations in our speed and steering models, we were able to get the runtime down to 0.29 seconds as a lower bound and 1.03 as an upper bound. The average runtime was 0.66 seconds and the standard deviation was 0.18 based on 150 trials. This meant that we increased our efficiency by more than 160 times.
We are proud of creating a working, intelligent system to solve a huge problem the world is facing. Although the system definitely has its limitations, it has proven to be adaptable and relatively robust, which is a huge accomplishment given the limitations of our dataset and computational capabilities. We are also proud of our probability of collision model because we were able to create a relatively robust, adaptable model with no training data.
We were also proud how we were able to create an app that compliments the drone. We were able to create a user-friendly app that is practical, efficient and visually pleasing for both customers and companies. We were also extremely proud of the overall integration of our drone with firebase. It is amazing how we were able to completely connect our drone with a full functioning app and have a project that could as of now, instantly be implemented in the marketplace.
What I learned:
Doing this project was one of the most fun and knowledgeable experiences that we have ever done. Before starting, we did not have much experience with connecting hardware to software. We never imagined it to be that hard just to upload our program onto a drone, but despite all the failed attempts and challenges we faced, we were able to successfully do it. We learned and grasped the basics of integrating software with hardware, and also the difficulty behind it. In addition, through this project, we also gained a lot more experience working with CNN’s. We learnt how different preprocessing, normalization, and post processing methods affect the robustness and complexity of our model. We also learnt to care about time complexity, as it made a huge difference in our project.
A self-flying drone is applicable in nearly an unlimited amount of applications. We propose to use our drones, in addition to autonomous delivery systems, for conservation, data gathering, natural disaster relief, and emergency medical assistance. For conservation, our drone could be implemented to gather data on animals by tracking them in their habitat without human interference. As for natural disaster relief, drones could scout and take risks that volunteers are unable to, due to debris and unstable infrastructure. We hope that our drone navigation program will be useful for many future applications.
We believe that there are still a few things that we can do to further improve upon our project. To further decrease runtime, we believe using GPU acceleration or a better computer will allow the program to run even faster. This then would allow the drone to fly faster, increasing its usefulness. In addition, training the model on a larger and more varied dataset would improve the drone’s flying and adaptability, making it applicable in more situations. With our current program, if you want the drone to work in another environment all you need to do is just find a dataset for that environment.
As for the app, other than polishing it and making a script that tells the drone to fly back, we think our delivery system is ready to go and can be given to companies for their usage with customers. Companies would have to purchase their own drones and upload our algorithm but other than that, the process is extremely easy and practical.
Try It out
- The Resiliency Challenge
- StanHacks 2
- Brother Hackathon Seattle
- TF 2.2 Challenge: Building AI Responsibly
- Cypher V½
- Silicon Valley Hacks
- Lumiata COVID-19 Global AI Hackathon
- Teens in AI COVID-19 Hackathon
- CovLab Global
- TOHacks 2020
- Quarantine Hackathon
- Safe Homes Access for Africa
- Hack at Home
- Cal Earth Hacks 2020: Hacking the Climate Crisis
- Community Hacks
drone, firebase, keras, opencv, python, swift, tensorflow, xcode