In this unique time of quarantine, the need for resources is more important than ever. With thousands of store aisles across the country completely cleaned out, the goods that we want and need are becoming more and more sparse. To keep our sanity, we need to know when these resources can be readily available for us so we can move on with life. To help win this race against time, I created "Ship It".
What it does
"Ship It" is not your regular e-commerce app. It scrapes the Internet to find the optimal shipping times that satisfy your wants as a user. By tying the item that you want and specifying an amount of time it needs to be delivered, "Ship It" will display to you many different options of the good you specified, all within the shipping time that you want. With a simple UI and UX, "Ship It" makes shopping during this stressful time as easy as possible.
How I built it
For the UI and front-end, I used React.js to easily render dynamic content and components. For the back-end, I used Python with Flask for the API and Beautiful Soup 4 for the web-scraping.
Challenges I ran into
The single biggest problem of the app was how long searching for an item took. Initially, to search for a single item, it took around 40 seconds to get an output of items. I struggled at first to find optimizations in my code, but after some look into concurrency and multi-processing, I reduced the response rate by almost 30 seconds.
Accomplishments that I'm proud of
While this search process still takes quite a bit of time, I am proud of figuring out ways to make the app more efficient. Efficiency in terms of Big-O notation and algorithms is talked about a ton in computer science curriculum, but no so much the efficiency of hardware and networking. Witnessing how much faster the search process took after concurrently running HTTP requests was incredibly satisfying, and I am super excited to figure out even more ways to make this process more efficient.
What I learned
I learned a ton about the web-scraping process and how Beautiful Soup works behind-the-scenes. I also got to reinforce my knowledge in back-end API development in Flask, as well as front-end development and managing state with React. Overall, this project helped me gain a stronger understanding of full-stack development, with the added spiciness of web-scraping.
What's next for Ship It
The next biggest challenge would be to make the scraping process even more efficient. With a search time of around 11 seconds, it is still relatively annoying to sit on a loading screen for that long. I look forward to looking into more about multi-processes and running more efficient searches across the web.
Try It out
beautiful-soup, flask, python, react