Personal Project Hygge
In Short
Finding a rental appartment in a crowded city is nowadays one of the most frustrating things of life. It takes time, it takes money, you’re vulnerable and at the end of the day you just want to live somewhere somewhat close to some of your friends in some sort of way. It’s really top of mind for a lot of people, me included for some time. Really, when meeting somebody new for the first time you can be sure that within 5 minutes either you or the other person has asked about the other ones living location and/or conditions, usually ending the discussion by joking that you can always rent a shoebox for 1000 euros per month underneath some picturesque little bridge. Life can be touch, but, I realized, why not automate the whole search procedure? I mean, we’re all in the same boat, searching and replying for hours on end until we found something affordable that was initially over-budget.
So, I decided to take matters into my own hands and start building scrapers to collect this data from the most important websites! Put a nice frontend on top, and things should go more smooth in the future! In fact, I spent so much time setting this up, in hindsight I perhaps should’ve spent that time actually searching for a house. But you never know how things go, again it thought me a lot, I had some fun. And a year later I actually moved! Partially thanks to what I call: “Hygge Housing”.
Hygge
So, Hygge, or, a tool to easily keep an eye on new real estate for rent in your city of interest such that you can act quick and get to rent that little palace you’ve always wanted. Hygge stands for that cosy, warm, comfy feeling you have when surrounded by good friends, a good ambiance, probably a glass of craftbeer or a glass of wine and often within one of those old cosy wooden bars without too much music inside. It’s a feeling hard to describe, but germanic and scandinavian languages usually have a word: you must know “gezellig” (Dutch), “mysa” (Swedish) or “Gemutlich” (German).
I am aware I might be a bit of language barbar here, perhaps there are some subtle differences. That’s the feedback I’ve gotten from some, but chose to ignore.
Similar Initiatives
In October 2023 I received an add from Stekkies. Seems like somebody else really pushed through and commercialized the idea. I’m curious how much work they have with keeping their scrapers up-to-date, if they even scrape at all, and when some of those rental websites will speak up. Since they might be losing out on income coming from ads.
Architecture
There are some similarities with Castor, but this time I went completely Kubernetes. Running everything in pods really changed the workflow a whole lot, since I no longer had to fiddle around within virtual machines but merely update github repositories and deploy those changes through ArgoCD. As such, I could iterate much faster! In the end, the architecture consisted of:
- ArgoCD for deployments
- Prometheus, Grafana, Thanos for metric-storage
- Azure File Share for storing objects and mounting them within the pods
- The frontend and backend are both written in Django, so no Typescript, Angular and slow NPM builds this time
- Data storage using both MongoDB and Postgres
- Cronjob automation initially using Flower and Celery but later on through Argo Workflows
- Static files served through a separate static file server
- Routing, networking through Azure, Azure Application Gateway and Azure Private DNS Zones
- Documentation through Docusaurus, including Algolia word search
Next to that, I spent a tremendous ammount on gathering data from those rental websites in the first place. Building scrapers that are some KISS can be quite a challenge, I ended up with quite a complex python codebase based on both OOP and Functional Programming designs. Passing around functions in python can get messy, but you learn how useful proper logging, tracing and checkpointing can be. Purely out of a need to overcome your own bad coding. After working meticolously on it during the cold winter nights, I managed to cover about 30-40 rental websites.
Using It
Hygge is purely meant to automate the searching procedure. You still would need to write comments on those new rental listings, but the idea is that if you are there first, you are much more likely to be invited for the viewing. Personally, I was still figuring out my life in a new city and not continuous busy with looking for a new place as I was also out of town still a lot. Perhaps that did not really contribute to me finding my place in a new city. What really helped in finding a new apartment in the end was deciding to spent more money on it, rather than automating the search itself.