GitOps Hydration: Should it be called Rendering?

In the past I have been reminded plenty of times that working in IT generally comes down to managing complexity. Practices (whether they are “best” or not), standards and overall conventions really help with that. So does consistency. As a matter of fact, it becomes more and more important as you start to look into the organization and across teams and domains.

Language plays a fundamental role in this too. It is for good reason that “Ubiquitous Language” is a cornerstone of Domain-driven design, because if we are calling things differently it will be more difficult to share the same understanding of a system.

This brings it to the topic of this blogpost: why do we accept that the step where Helm charts or Kustomize overlays are turned into concrete Kubernetes manifests, is not called rendering but Hydration?

To me, the term feels wrong, but it is probably here to stay.

Mechanistic Rendering

In fact, rendering existed long before GitOps cared about it. It is about evaluating templates with inputs to produce concrete output.

Meaning:

• In Helm: Templates + values -> YAML
• In Kustomize: Base + overlays -> YAML

As such, rendering is also not that interesting in this case. It merely allows for engineers to work with the underlying manifests through their own abstractions. The rendering itself is merely an implementation detail, rather than an architectural boundary where responsibilities, guarantees and ownership change.

However, as GitOps practices matured and the abstractions over the underlying Kubernetes manifests increased, teams began to struggle in managing the underlying complexity effectively. Pull requests and cross-environment resource management focuses on the top abstraction, whereas the resources on the cluster itself would determine the final result. In many cases, this led to teams rendering their abstraction into the underlying manifests and inspecting those!

Much of the initial though leadership around some of these issues can be attributed to the friendly folks at CodeFresh. There is a lot to learn from that corner, and I would like to draw your attention to a couple of specific pieces. Namely, this primer on Gitops, this arcticle on the pains of GitOps 1.0 written in 2020, this article on not using a branch per environment written in 2021 and this article from 2022 what to do instead. Bear in mind that the company is invested in selling you solutions for some of these problems, as such it is in their best interest to explain the problem very clearly as well as some solutions (irregardless of how challenging the implementation would be). Early January 2023, they are the first to introduce pre-rendering manifests in a second Git repository, which also contains these two example repos: Git repo 1 and Git repo 2 as well as the image below.

Hydration as Workflow Boundary

These articles have been very successfull in shaping some anti-patterns and best practices around GitOps throughout the industry, especially around the usage of ArgoCD. ArgoCD has been showed over time to be just solid. But once you start setting this up and scaling things out, there are several known hurdles. How should you be dealing with mult-cluster setups? How can you easily track which versions are deployed where? How to deal with gigantic and abstract diffs in production?

More recently some of these notions have become more mainstrain, which is large thanks to Akuity, this particular Akuity blogpost on the Rendered Manifests Pattern (see also this talk or this another talk also by somebody working at Akuity) and this blogpost on the Kubernetes Shift-Left Philosophy. The last blogpost in turn refers to this KubeCon EU 2024 Conference Talk (which I could have seen but deliberately decided to skip back then, well…).

Perhaps less known is that the people behind Akuity are some of the fouding engineers of ArgoCD (Hong Wang, Jesse Suen, and Alexander Matyushentsev), who are also behind Argo Workflows, Argo Events and Argo Rollouts. In this proposal, some other noteworthy contributors first introduced the concept stating that:

“Many organizations have implemented their own manifest hydration system. By implementing it in Argo CD, we can lower the cost to our users of maintaining those systems, and we can encourage best practices related to the pattern.”.

In this case however, rather than keeping with “rendering”, ArgoCD has directly coined the term hydration and from there on the term seems to have stuck. In other parts of IT, hydration has had a slightly different interpretation. In React, for example, it means attaching runtime behaviour and state to an already-existing structure. That’s not really what is going on with ArgoCD Source Hydration, since the actual deployment of the YAMLs still follows the same process as before as it is directly executed by ArgoCD. In this case, however, ArgoCD has also come to be involved in the creation of the repository itself on which the manifests depend.

Perhaps it is intuitive, but for the nitpicker in me it is also imprecise. Anyway, it is what it is. The term seems to stuck, and truth be told that’s not necessarily a bad thing since it does account better for the workflow boundary, rather than the mechanistic operation. Perhaps we did need to repurpose an existing concept for this after all. Renaming it now would mean more inclarity, since many of blogs, podcasts, talks have already adopted the term.