Why This Matters Beyond Engineering

The shift from CI/CD-based infrastructure management to API-first control planes isn’t just a tooling choice, it’s an operating model decision. Organizations adopting control plane patterns like Crossplane gain continuous drift detection, self-healing infrastructure, and a declarative interface that can be governed centrally while giving teams autonomy. That translates directly into reduced operational risk from configuration drift, fewer incidents caused by manual changes, and a foundation for intelligent operations (think AI-assisted compliance and anomaly detection) that only becomes possible when infrastructure is managed through real-time APIs rather than batch pipelines.

Introduction

In this blogpost, we are diving deep into K8s Controlplanes amd demonstrate how each progression of complexity can be brought back to the more generic MVC Software Architecture Pattern. That is, the Model-View-Controller Pattern.

Why this blog might be interesting for you? You’re trying to make sense of API-first infrastructure management, real-time-operating controlplanes and the future of IaC.

What is MVC?

The MVC design pattern separates an application into three main components: Model, View, and Controller, making it easier to manage and maintain the codebase. It also allows for the reusability of components and promotes a more modular approach to software development. See also the screenshot below.

When MVC matters

• Decoupling user interaction from the underlying features and the business logic
• Reusability of underlying components across the business, through modularity and testability

For simple applications, MVC patterns can often bring too much unnecessary complexity.

If you don’t believe me that’s fine. However, I asked ChatGPT for a list of very large codebases that conform to the MVC pattern as well, and it basically threw up every large social media platform you can think of (Linkedin, Youtube, Instagram), as well as Shopify, Github and Netflix. Also note that a slight modification is also very commmon in many webframeworks such as Hugo or Django (Model-Template-View).

Going from MVC to the K8s System

Kubernetes has maybe one of the best documentation of any piece of open source software. Very useful if you’re figuring out the Kubernetes mechanics, or debugging some of your deployments. So for a proper deepdive, there’s really no better place than those docs. For completeness, I’ve added the below image.

Looking at this diagram, as well as the MVC one before, we can make the following conceptual mappings.

Model

In K8s this would be the state of the cluster, its resource objects, interobject relationships as well as the statuses of the individual objects. These objects are generally represented as manifests.

In other words, the model represents the desired as well as the actual cluster state. Within MVC, you generally refer to this as your ORM entities or your data model.

View

This is how users see and interact with the cluster. This is mostly centralized around the central API server. Note that the API-server in this case mostly helps to retrieve information either from its cache or from the peristed storage. It can also be extended by more custom observability on a pod-level. Like for example through Prometheus or Opentelemetry.

Controllers

There’s no such thing as one single controller. Rather, it is distributed across the API Server, which acts as the frontend, K8s-native controllers such as the Replicaset controller or the Job controller, as well as the Kube-scheduler.

Therefore, together with the model and relationships between the objects, the controller implements the business logic required to move between entity states.

Reconciliation

Note that MVC-patterns rely on request-response cycles and are expected to implement changes atomically in the background. Kubernetes does a lot more than that. It is specifically designed to allow for object deployment/creation failures and to actively reconciliate afterwards.

By continuing this line of though you quickly start seeing more of a divergence between MVC as pattern and K8s as an orchestrator. MVC patterns say nothing about how users interface with the system as a whole, whereas K8s generally works with declarative state + reconciliation. If you want to be very exact, you could also say that you can even work with the K8s APIs in an imperative manner (e.g. delete, apply, etc all manually) or that the underlying controllers work with the resources in an imperative manner. But then you’re really zooming in on a specific parts of the system as a whole. Hence, people often refer to this as Declarative for Humans, Imperative under the hood.

K8s Deployments

So, what does this actually mean, let’s break it down a bit more.

When you apply a Kubernetes manifest to a cluster, you actually send it to the API server as a model/desired state in etcd, controllers will pick it up and start moving the state of the cluster towards it.

For example, if your manifest contains the definition a Kubernetes deployment including 2 pods, the API Server will notify the Controller Manager, which notifies the ReplicaSet Controller to create a Replicaset, which in turn is an abstraction over a deployment. The ReplicaSet is essentially a pod selector that matches a label selector.

The unscheduled status of the underlying pods trigger the Scheduler, which will try to attempt to find or create a node matching the criteria for the pod. Once that node has been made available, the ReplicaSet controller creates the pod using the local kubelet on the node. If there’s a failure during the rollout, it will either perform a rollback or retry again.

Note that everything that is related to this specific deployment, follows similar parallel processes. Volumes, Configmaps, Secrets, adding nodes to the cluster, etc, etc. All seemingly together!

K8s Operators

Let’s start extending these concepts now. Note that I rely on the official documentation and the original white paper as well as my personal experience.

From that whitepaper:

You must already see where this is going, but the whitepaper also providers some more details.

Actually, we have a different abstraction with different names, but it’s just MVC again! s Building upon the earlier diagram, below you see an overview of the Operator pattern.

As such, the Custom Resource Definition allows you to extend the K8s Model by extending your own custom resource types. This means that you can leverage the native K8s tooling to work and interact with these resources, such as kubectl, the Kubernetes API Server, etc, etc. It thus both extends the

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # name must match the spec fields below, and be in the form: <plural>.<group>
  name: crontabs.stable.example.com
spec:
  # group name to use for REST API: /apis/<group>/<version>
  group: stable.example.com
  # list of versions supported by this CustomResourceDefinition
  versions:
    - name: v1
      # Each version can be enabled/disabled by Served flag.
      served: true
      # One and only one version must be marked as the storage version.
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                cronSpec:
                  type: string
                image:
                  type: string
                replicas:
                  type: integer
  # either Namespaced or Cluster
  scope: Namespaced
  names:
    # plural name to be used in the URL: /apis/<group>/<version>/<plural>
    plural: crontabs
    # singular name to be used as an alias on the CLI and for display
    singular: crontab
    # kind is normally the CamelCased singular type. Your resource manifests use this.
    kind: CronTab
    # shortNames allow shorter string to match your resource on the CLI
    shortNames:
    - ct

Note that the CRD does not implement any logic or behaviour. Neither does it reconcile desired state with actual state. It merely allows the cluster user to work with these new resources in K8s fashion.

Let’s say I create a resource according to the above CRD:

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * * */5"
  image: my-awesome-cron-image

Great, now this will be part of the cluster state. However, without some sort of Controller translating that into actions, there’s not much happening. You essentially added a record to the central ETCD, because the controlplane does not really have any instructions to do something with this type of resources.

This is where a custom custom controller of the Operator comes in. You see, the Operator Pattern is just MVC. This custom controller then translates the custom resource in K8s-controlplane-native resources, such as Deployments, Services, Secrets, Ingresses, etc, etc. These new objects are then recognized by the K8s controllers, and a similar loop as described above is triggered. (ReplicaSet Controller, Scheduler, Kubelet, etc).

You might say that the Crontab-example is just silly, since K8s natively already provides these capabilities. Sure, but this pattern is much more common than you might think.

Like for example the Certmanager Operator:

Or the Prometheus Operator:

Or any of the 400 community operators you can find at the Kubernetes Community Operator Hub.

Crossplane

So you have the central K8s Control Plane, and the standard Operator pattern to extend that control plane for more custom K8s objects. Note also that we can link cluster-external resources, like for example a “Let’s Encrypt”-certificate, back to resources on the cluster using the underlying controllers.

It’s a pretty powerful idea actually, and not too uncommon as well. In fact, there are many more of these examples and if you are using any type of Public Cloud based Kubernetes cluster. E.g. a controller that handles external DNS, Ingress or even API gateways, as well as nodes. So you might already be heavily depending on it!

The Prometheus Operator also shows that you can implement quite some complexity in that central controller. You can in theory stuff everything you need in order to deploy your cloud infrastructure into a single controller!

This might be a particular good idea if you really need fine-grained control over reconciliation cycles and external integrations, but you also generally would require your team(s) to be highly mature in their application development and management processes. The controller would become very central in your infrastructure, and potentially limit flexibility, reuse and modularity in the long run. Platform teams, nonetheless, work generally in a different fashion as well as rely more on Kubernetes and GitOps.

In fact, Crossplane’s bet is that most platform teams prefer declarative blueprints over writing and maintaining custom controllers. But for very complex workflows, operators might still make sense since expressiveness can be quite limited and the complexity is essence move to the YAML. Next to that, coupling in general becomes more implicit and it’s often more difficult to debug these declarative pipelines. Tooling has gotten better for this too though.

How does that bet manifests itself in Crossplane? Rather than pushing more logic into the controller itself and further extending the options on the one custom resource in front, it extends the concept of a custom resource into a composition of resources and their interdependencies. This way, not only the resources but also parts of their dependencies thus become available to the one that defines the definition of that composite resource, the Composite Resource Definition or XRD.

You might want to read that segment a couple of times, things are really getting more abstract here. So I have a XRD, which defines how a Composite Resource looks like as well how the Kubernetes API should make them available to the users in the View-layer as a “Claim” (in Crossplane V1, for V2 you generally create a Resource Composition directly). So far, we are only scratching the surface of the Model & View layers. To make the translation to actual “Managed Resources”, Crossplane makes use of a “Composition”.

Crossplane & Operators

If you have gotten this far in this article, props for you! Now we have waded through most of the definitions, let’s go full-circle and take those brains for another spin!! You can also look at Crossplane as a meta-operator framework consisting of not one but two Operator pattern implementations!

• Crossplane Core acts as an operator for Composite Resources. It watches XRD-based resources and reconciles them into multiple managed resources according to the composition. This part handles the high-level orchestration and abstraction without too much custom code. Note that this thus only involves K8s-based resources, rather than external to the cluster. You could compare it to the Prometheus Operator example.
• Crossplane Providers in turn install CRDs for Managed Resources and run controllers that reconcile those resources with external APIs. Just as the CertManager example would do.

Crossplane Community

Note that there is considerable backing behind Crossplane as well, further proving that the underlying problem is real.

In this presentation at KubeCon EU 2024, it is mentioned that:

• 5+ years development
• 1950+ total contributors to the project (top 10% of all CNCF open source projects)
• This includes: almost 350 companies, and 130 companies that also have a maintainer on board
• The steering committee has members from: Nokia, Apple and Upbound
• Adopters at scale: Nike, Autodesk, Grafana, NASA, Akamai, SAP, IBM, VMWare Tanzu, …

Next to that:

• There is official tooling to help you build your own Crossplane Provider from any Terraform Provider.

Other Things To Know

To alleviate memory pressure on the central Kubernetes APIServer, Crossplane 2.0 introduces Managed Resource Definitions. Storing configuration in a database, rather than as code, used to be a bit of an anti-pattern. Even though Crossplane still implements this under the hood, albeit much more user-friendly and standardized (given that you know and understand K8s well), this new feature allows the CRDs to only “be expressed” when they are required for the creation of a Managed Resource.

With each CRD consuming 3 MiB of API Server memory, partially due to the creation of API Cluster endpoints, for large providers (100+ CRDs) this can really make big difference!

Declarative Day-2 Workflows

Crossplane V2.0 also brings options to further standardize operations for your underlying resources in a Crossplane-native way. In the past, you would do this by making use of K8s (Cron-)Jobs, but nowadays you can do so through Operations. Useful for tasks like backup&restore, rolling upgrades or validation steps.

Crossplane V2 & Claims

Right, so. Readers that have been paying attention, or that are reading this several years after writing, might already have stumbled over the “Claim” object in an earlier diagram. Well, this object is actually regarded as rather awkward, as it mostly facilitates namespacing of multi-tenant clusters. By namespacing the Composite Resource itself, namespaced users can work with the resources directly within their namespaces without either needing larger cluster access or bothering cluster admins to help them debug their Managed Resource deployments.

Note that it is also still possible to create cluster-scope Composite Resources.

Crossplane Functions

Wait there is Functions too? That was my reaction too, but they’re actually pretty clever. When you’re writing your own Composition, you do not want to have to reinvent the wheel every time you run into use cases that require advanced logic. These functions allow you to simplify the Composition, and rely on either community, custom or Upbound-build Functions. And there are a lot of them here, like for example:

• Patch & Replace,
• Tag Manager, to manage resource tags
• A function to run Shell commands
• Automatic Ready Detection
• Environment configs loading
• Go Templating
• Function for querying the Azure resource graph, as well as msgraph

Plenty of options, take a look for yourself!

And some more exotic, but potentially extremely useful ones:

• Automated EKS Discovery and integrating it with a central ArgoCD -> Making multi-cluster initialization much easier!
• A function to sollict a manual approval (UXP only)
• A function to integrate Claude with your Composition (OpenAI also exists, both UXP only)

It’s not clear how much work it is to create one of these functions though. Some of them do not seem to have a lot of love on Github, or seem to only be supported by Upbound.

Upbound & AI-native Controlplanes

Crossplane was created by Upbound in 2017, and donated to CNCF in 2020. Even though Crossplane is currently open source, with a large variety of contributors, Upbound is clearly its primary maintainer. They have recently donated official providers for AWS, GCP, Azure and related code-gen tech to CNCF for vendor-neutral governance, but they also offer an Enterprise-grade distribution and have Venture Capitalist backing. It should not come as a surprise that Upbound Crossplane (or UPX) has a strong focus on what it calls “AI-native Controlplanes”, and is even pushing for AI for intelligent operations (through compliance checks and anomaly detection) with its 2.0 release.

It remains a bit of a visionary product, with many organizations more relying on CI/CD-based solutions, rather than API-first. And because it is part of CNCF, there is some mitigation against vendor lock-in and push to keep Crossplane community-driven. We have off course seen more open-source projects develop a more commercial angle and eventually pivot, so you maybe never really be sure.

AI-native Control Planes

Personally, I have to look into this a bit further to see how this would work. Upbound is painting a future where human operators work side-to-side with autonomous agents to debug issues in existing clusters and push fixes. Some of the recent changes also work towards that, like for example the formalization of the Crossplane Operation. Upbound has provided a short demo, that mostly relies on deploying either Claude or OpenAI directly in the cluster, and use MCP to effectuate changes against existing Crossplane providers. A pretty powerful idea, albeit limited by the known constraints around AI and its indeterministic behavior which might be a unwanted feature when dealing with infrastructure that should behave reliable. But there are already some use cases that might benefit from this approach without sacrificing reliability, such as: Audit & Compliancy, first-line debugging of issues and information gathering, as well as learning about performance patterns and providing small tweaks at runtime.

Again, I am planning to look into this a bit further. Stay tuned…