Skyworkz Volksbank
In Short
At Volksbank I fulfilled the rol of DevOps engineer within the Release Engineering team. Together, we were responsible for many of the tooling around DevOps and CI/CD, most importantly BitBucket, Jira and Jenkins.
The Volksbank
Since the Volksbank actually consists out of 4 smaller banks (SNS, ASN, RegioBank and BLG Wonen), and has been nationalised by the Dutch government in 2013 as a late consequence of the financial crisis of 2008. This also explains the strong focus of the bank on on human and society, notwithstanding that the Volksbank is still a bank in the end. Personally, I always appreciated how high the bank scored on sustainability indices and sustainable finance/investments, although I also found it difficult to assess those statements.
Technologies
We relied as much as possible on AWS and EKS, but here and there we were tied to on-premise VMs due to legacy restrictions. Those linux VMs tended to give most of the problems we encountered and sometimes required manual patching to prevent the services running on those machines from crashing. For the rest, Java is the preferred language within the Volksbank and we therefore stayed quite close to that in the sense that we hosted our own BitBucket, Jira and Jenkins internally as well as some SaaS-tooling for automated deployments and releases to VMs. The most challenging components tended to be BitBucket, due to the organization being highly dependent on it (~800 developers, daily) and Jenkins, which was running in the form of several different instances. Some of them on on-prem VMs and the newest Jenkins environment on EKS with dynamic slaves, and an extremely fat master pod.
My Work
As a member of the team, I was responsible or involved with various different things:
- Migrate several Jenkins instances from on-prem to EKS, including containerization and using kustomize
- Improve the observability of all our Jenkins-on-EKS instances through a custom Jenkins prometheus plugin, improved dashboarding and job queue tracking.
- Improve triaging of Jenkins and EKS auto-scaling issues. There were various incidents that taught us how important it is to make sure your Kubernetes nodes are larger than your pods, because of conflicts between seemingly independent schedulers.
- Upgrade the Bitbucket instance several major versions without downtime
- Implement BitBucket backups to AWS S3 from the on-prem VM through Bash scripting
- Lots of bash and python scripting to glue steps within CI/CD jobs together
- Trainings in AWS and Terraform, as well as providing advice in various discussions around having people standy versus investing in the availability of your services and the impact on burn-out rates within your organization.
Joining a large organization like the Volksbank as an interim external consultant/engineer was challenging in many ways, but also very interesting. After having been at KPMG for several years, and advising corporates and startups on many different aspects of IT, data and cloud, I was happy to actually join one for once and experience how it is to work for such an organization directly. As an external advisor you are still mostly going to those places where they are in need for some kind of help or advice, but being internally part of such a place did teach me a lot empathy. It can be very challenging to deliver modern CI/CD-services while making use of new technologies within a highly regulated environment given that you have about 800 daily customers who both know exactly where to find you within the organization as well as how to put pressure on your team. At the same time, you see the technological diversity and you learn to appreciate being pragmatic. It’s important to guard your scope and you can be overly dilligent in some scenarios. Not everything needs continuous testing and continuous deployments. Somethings might not even need version control. And definitely not every team can be improved by being forced to work scrum, agile, kanban or anything in between.