Skip to main content

Architecture

Overview​

Organizations adopting Kubernetes at scale typically end up managing multiple clusters, whether divided by environment, tier, team, or business domain, largely due to the weak isolation provided by standard namespaces. In this traditional model, every single cluster carries its own independent stack. This includes everything from the control plane to storage (which requires at least three dedicated nodes to satisfy the etcd quorum), management applications, service meshes, dashboards, and observability tooling. Managing this overhead independently requires a continuous cycle of upgrades, health checks, and maintenance. Scaled across dozens of clusters, this model introduces a massive tax on both your infrastructure budget and engineering velocity.

Heir takes a different approach. Rather than treating the control plane as a heavy, fixed piece of infrastructure for every cluster, Heir runs the control plane components as a single, stateless pod inside a shared management Kubernetes cluster. This fundamentally shifts the availability problem: platform engineers only need to solve for control plane availability once when designing the management cluster. Every additional cluster provisioned on top automatically inherits this availability, entirely eliminating the need to provision or maintain duplicate infrastructure.

Heir cluster architecture

Heir is split into two distinct planes: the control plane, which runs inside the management cluster, and the data plane, which runs on worker nodes. The two communicate exclusively over connections started by worker nodes.

Control Plane​

The Heir control plane packages the standard Kubernetes control plane components (API server, scheduler, controller manager, and the communication proxy) as a single stateless pod running inside a management Kubernetes cluster. The management cluster can be any conformant Kubernetes distribution, including managed services such as EKS, GKE, AKS, or DOKS.

Heir control plane

Running as a stateless pod has a fundamental implication: the control plane is no longer a special snowflake that requires dedicated infrastructure and specialist knowledge to keep alive. It is just another workload. Kubernetes already knows how to schedule, restart, and reschedule workloads, and it does the same for Heir control planes. State is never stored in the pod itself; it lives in an external storage backend, so the pod can be terminated and restarted on any node at any time without data loss.

This also means availability is no longer a per-cluster concern. When a new tenant cluster is provisioned, its control plane inherits the availability guarantees of the management cluster without any additional infrastructure. A management cluster backed by a managed Kubernetes service with a multi-zone control plane gives every tenant cluster running on it the same resilience, for free.

Process Model​

All control plane components run as separate processes inside a single container, managed by S6 Overlay. The container environment is configured so that no process writes local state (beyond logs), meaning, the principle is that faults are always a consequence of not satisfying external dependencies, it also means the container can be stopped and restarted at any time without disruption.

One of the oft-repeated Docker mantras is "one process per container", but we disagree. There's nothing inherently bad about running multiple processes in a container. The more abstract "one thing per container" is our policy - a container should do one thing, such as "run a chat service" or "run gitlab." This may involve multiple processes, which is fine.

Cluster Agent​

While the Runtime controller is responsible for provisioning a tenant cluster's control plane, clusteragent is what keeps that cluster running once it's bootstrapped. If the Runtime controller is the shipyard, building the ship once, launching it, and coming back only for major refits (cert rotations, upgrades), clusteragent is the ship's captain. It runs as its own Deployment in the management cluster, but acts directly against the tenant cluster it serves, handling everything the control plane can't do for itself: approving kubelet-serving certificate requests, processing worker nodes' requests to join the cluster, syncing Kubernetes Service EndpointSlices, and running the Helm Controller that installs and manages addons. Heir provisions exactly one clusteragent per tenant cluster.

Operators and Addons​

In standard Kubernetes, management tools and operators (like Istiod or ingress controllers) run inside the very clusters they manage. This means every single cluster needs its own separate copy of these tools, wasting resources and tying their uptime directly to your application environment.

Heir flips this model. With Heir, these management tools run centrally in a shared management cluster. They control the tenant clusters remotely through API server. For example, a single service mesh deployed in the central cluster can manage traffic across multiple clusters simultaneously, leaving zero footprint on the actual worker nodes.

This separation has meaningful consequences:

  • Worker nodes are reserved exclusively for application workloads
  • A single operator instance can govern multiple data planes simultaneously (if supported), reducing duplication
  • Operators vulnerabilities are patched once on the management cluster
  • Addons can be upgraded centrally without touching worker nodes

Data Plane​

The Heir data plane is distributed as a single self-contained binary that packages everything needed to run a Kubernetes worker node, except the Init System, iptables, and Linux kernel modules. It runs on bare metal, virtual machines, containers, or any Linux host.

Worker Profile​

The worker-profile ConfigMap in kube-system is the single source of configuration distributed to all worker nodes. It carries:

  • Control plane endpoint address
  • Kubelet configuration
  • CNI provider details and more

The worker process caches the profile locally after reading it, so kubelet can be restarted cleanly even if the API server is temporarily unreachable. The heir worker process continuously watches this ConfigMap for changes, it is this reconciliation loop that enables automatic cluster operations such as kubelet configuration updates and CNI reconfiguration without manual intervention or node re-provisioning. Any update to the Runtime spec that affects worker configuration is reflected here and picked up by nodes on their next startup.

Bootstrap Tokens​

Nodes register using a short-lived bootstrap token generated by an administrator (using WorkerJoinToken). The token grants its bearer the minimum RBAC permissions necessary to submit a Certificate Signing Request. Once TLS bootstrap completes, the token is no longer used and the node authenticates exclusively with its signed client certificate.

When heir worker starts, it uses the bootstrap token to submit a CSR to the API server requesting a kubelet client certificate identifying the node as system:node:[hostname]. The Cluster Agent watches for pending CSRs and approves them only after verifying that every SAN in the request matches an address already registered on the corresponding Node object. This prevents a compromised token from being used to obtain certificates for arbitrary node identities.

Network Model​

All connections flow from worker nodes to the control plane. The control plane never initiates connections to workers. Requests from the control plane to workloads on workers (such as kubectl exec or log streaming) are tunneled over the same outbound connection established by the worker through multiplexing.

This design has practical consequences:

  • Worker nodes can sit behind firewalls or NAT with no inbound rules required
  • Heir works in air-gapped environments without internet access after initial setup
  • Heir is well suited to edge computing deployments where workers are in remote or constrained networks
  • Control planes can run on managed cloud Kubernetes services while workers run anywhere

Multi-tenancy​

Because each tenant cluster is a single pod in the management cluster, running multiple tenant clusters is as simple as deploying multiple Runtime resources. Each gets its own isolated control plane, API server, and kubeconfig, while sharing the management cluster's infrastructure. This makes Heir an efficient model for multi-tenant Kubernetes at scale.