To get food, I learned Selenium and web scraping in one day. And I’m not even a programmer.
Goal: Buy groceries. In simple caveman’s terms: “Get food”.
Problem: You know, the virus. I don’t want to go grocery shopping and wait in line to get into the store.
Solution: Order groceries online.
Another problem: The nearest delivery day is more than 1.5 months away. Not good.
Possible solution: Sit and refresh the page for hours, hoping that some slots free up and get back into the pool. Then, hope to be quick enough to snatch one of the slots.
Better solution: Remember that you can do anything in Python. Remember that I saw some useful stuff in an aptly titled book recently: “Automate the Boring Stuff with Python”. Go through the book, quickly learn to control the browser, write a program that checks everything automatically and reserves the nearest delivery slot as soon as it becomes available.
Details:
It was a few weeks ago. When I was failing at my main mission to bring food to the table, I came up with the idea above. I started working on it in the morning. I had to start pretty far back, going through the book and learning the first steps to control the browser with Python.
It was April 17th and the nearest delivery slot was for June 4th.
At first, I wanted to do something simple, similar to what Utku described in his article I had to build a web scraper to buy groceries. But in my case, it wasn’t possible. Setting session cookies and firing GET requests didn’t work. It had to be the whole shebang: with the browser, logging in, entering all data and simulating all the user actions.
I had the first working version around 2 AM and was about to go to sleep but then I decided that I need to automate it and make it check the availability every minute, while I would be sleeping.
I had two criteria to decide if the program should claim the slot automatically: it has to be earlier than June and the time of delivery has to be later than 12 PM. I didn’t want to be woken up at 6 or 7 AM. With that finally automated, I went to sleep around 6 AM, having coded since the morning of the previous day.
While I was sleeping, around 9 AM, the program found and quickly claimed the slot that opened up. But it was for May 23rd, more than a month away. It wasn’t good enough. When I checked the results in the morning, I canceled the reservation and modified the date to be in April. May was still too far away. Meanwhile, the nearest free slot was again for June.
15 minutes later, while I still didn’t get a chance to get dressed, the program did what it was supposed to do: it found and booked the new slot. For two days later! For a fitting time: 12:30 PM.
The moment of triumph:
A quick summary of what the program did:
- Open Chrome and go to the store’s website
- Close the welcome pop-up
- Click the login button
- Enter username, password, click “Login”
- Check if logged in correctly
- Check the nearest delivery date
- If it’s still in June, add a line to the “grocery-log.txt” file with a timestamp and a “Fucking June” annotation, then exit
- If the nearest delivery date is in April, check if the delivery time is before or after 12 PM
- If before, exit and start again in a minute (I didn’t want to overload the servers by creating too many requests)
- If after, click a few buttons to first select then to confirm the reservation. Done.
Technical details:
MacOS, Python 3.7.3, Selenium, crontab.
Things to improve for the future, if I were to use it more than once:
- Headless mode, to avoid launching the browser every minute and to be able to do other stuff while the program runs quietly in the background.
- Moving it to a server, so that it doesn’t have to run on my computer.
- A few more rules for choosing dates and times for maximum flexibility.
Aftermath:
I shared a quick write-up on my Facebook profile, thinking that it could be interesting or funny for a few people.
Instead, it became my most-liked, most-shared (400+ shares in total), most-commented post, beating everything else by an order of magnitude.
And I was able to do all this from scratch, while not even being a programmer! I was sure programmers were automating such stuff left and right. Turns out, very few of them did.
The grocery chain’s representatives even got in touch with me, congratulated on the inventiveness and said “By the way, the board of directors says hi.” That was certainly unexpected :)
In Facebook comments, the grocery chain’s representative said they do free some slots for the near dates, at some specific hours. They were sure people knew that but nobody did.
A week or two later, when I visited the website again, I saw the information about the times when they free up slots plastered all over the site: as a huge pop-up on entry, as an info bar at the top, as a notification in their app and they even sent a broadcast email to all their customers with this info.
