How to get a room in Berlin scrapping Wg Gesucht with NodeJS
You can check the serverless script alternative here.
Berlin is a great city to live in if you are a software developer. It has an immense amount of tech startups and it’s quickly becoming one of the most important tech hubs in Europe.
But it also has a great problem. It’s almost impossible to find a place to live. Depending on who you speak with the average time spent in finding a room or a flat is about 3 months.
There are several alternatives and websites to find a place to live in Berlin, such as:
- Nestpick
- ImmobilienScout24
- Immowelt
- eBay Kleinanzeigen
- Wg-gesucht
- Wg-suche
Each of them has a specific target and flat profile. In my case, I was focusing on Wg-gesucht since I found it to be the most valuable for the shared apartment alternative.
Automating the process!
I decided to create a NodeJS app to automate the whole process. The app was initially running from my computer but later on, it was deployed into a free tier VM of google cloud: https://cloud.google.com/free/
How does it work
- List the rooms
After configuring the app, every 5 minutes a search is performed using the provided filters. The list is processed using cheerio for filtering and crawling purposes.
2. Get the language of the Ads
The chance of getting the room increases by writing a message to the owner in the correct language. The app uses the franc module in order to determine which language was used.
The rooms are then stored in a db.json file. Since the scope of the project is small, a json db will do the trick.
3. Send a message to the owner
Every 5 minutes the db.json file is processed and the messages are sent to the owners. If the message is delivered successfully then the db is updated.
Results
The week after starting the script I was invited to see at least 20 rooms. In less than two weeks I have already rented the room I wanted.
The app
The script can be found here: https://github.com/juancruzmartino/wg-gesucht-scrapper
