Helm-based cluster addons​

Up to this release, addons like CoreDNS and the CNI plugin were provisioned from manifests rendered by the control plane, which meant the Runtime CRD grew a bespoke, typed field for every setting each one needed. That doesn't scale past a handful of addons, so it's been replaced with a Helm controller instead. spec.cluster.addons.helm now lets a Runtime manage its own addons:

spec:
  cluster:
    addons:
      helm:
        charts:
          - name: coredns
            enabled: false
          - name: cilium
            install:
              chart:
                name: cilium
                repo: https://helm.cilium.io/
                version: "1.15.0"
          - name: metrics-server
            override:
              values:
                content: |
                  replicas: 2

Use enabled: false to drop one of Heir's built-in addons, install to add an entirely new chart to the bootstrap phase, or override to tweak a built-in addon's Helm values without touching its chart source. Each entry is keyed by name.

A dedicated clusteragent​

Everything that needs to reach into the tenant cluster, rather than just manage the management cluster's view of it, now lives in its own component: clusteragent. That's worker join tokens, kubelet-serving CSR approval, Kubernetes Service EndpointSlice sync, and the new Helm addon/HelmChart reconciler described above. Splitting it out keeps the management-cluster controller manager focused on the management cluster (guarantee that the cluster is healthy), and gives the tenant-facing half its own image, CRDs, RBAC, and release cadence.

WorkerJoinToken and HelmChart move clusters​

This is the release's breaking change: WorkerJoinToken and HelmChart moved out of the controlplane API group into a new one, clusteragent.tardigrade.runtime.io/v1alpha1, and are now reconciled against the tenant cluster, not the management cluster. WorkerJoinToken is also cluster-scoped now, and no longer carries a runtimeRef, it's always bound to the cluster it's created against, so there's nothing left to point at:

apiVersion: clusteragent.tardigrade.runtime.io/v1alpha1
kind: WorkerJoinToken
metadata:
  name: new-node
spec:
  ttl: 3h

If you have any tooling that applies or reads either resource under the old controlplane group, or against the management cluster's kubeconfig, it needs to target the new group and the tenant's own kubeconfig instead. See the updated WorkerJoinToken reference and Quickstart for the new shape end-to-end.

The typed cluster.network.coredns and cluster.network.cni Runtime fields are gone as a consequence, both are provisioned exclusively through spec.cluster.addons.helm.charts now, and if you were using the previous map-keyed spec.cluster.addons shape, it's been restructured into the charts list shown above.

CLI cleanup​

heir build and heir provision controlplane have been removed. They were superseded a while ago by the goreleaser/Dockerfile image pipeline and the Runtime CRD controller respectively, and had no remaining callers. heir delete controlplane is unaffected. Separately, if you import Heir's Go packages directly: api/v1alpha1 and internal/controller moved to api/controlplane/v1alpha1 and internal/controller/controlplane as part of migrating to kubebuilder's multi-group layout, which is what unblocked adding the clusteragent group in the first place.

Fixes worth knowing about​

• PKI SAN rotation now rolls control-plane and plane-tunnel pods. Previously only the PKI Secret was regenerated; running pods kept serving stale certificates until something else happened to restart them.
• WorkerJoinToken no longer rewrites its status on every reconcile, which had been bumping resourceVersion and re-triggering its own watch in a tight loop.
• containerd and its containers now survive a heir restart. The generated systemd unit used KillMode=control-group, which signalled every process in heir's cgroup, containerd, shims, every running container, on stop or restart. It's now KillMode=process with Delegate=yes, so only heir's own PID is signalled and kubelet/containerd keep managing their own per-pod cgroups.
• Removed kubelet's --runtime-cgroups flag, which pointed at a containerd.service cgroup that never exists under heir's process model.
• heir provision worker no longer fails with open ... text file busy when re-run against an already-joined node.

containerd config schema v4​

The generated containerd config migrated to schema v4, a requirement from containerd 2.3+, which splits the CRI plugin into separate images/runtime plugins. The generated config.toml also now imports /etc/lib/heir/containerd/conf.d/*.toml, so you can layer extra config, a registry mirror, for example, in a separate file that survives heir provision worker regenerating the main file on every run.

Local development with docker-compose​

Local worker nodes are now docker-compose bootloose containers instead of Vagrant VMs, mirroring the same image, privilege, and cgroup setup the e2e test helper already used. A new .local/run-locally.sh script drives the whole walkthrough end to end: build, stand up the management cluster, deploy the controller manager, provision a tenant Runtime, wait for clusteragent to come up, mint and deliver join tokens, join both dev-worker0 and dev-worker1, and confirm both reach Ready.

Install the Operator with Helm​

Heir is moving from a CLI-driven workflow to an Operator-driven one, and this release is the first concrete step, not just a change in how you happen to invoke commands. The wider vision is for every piece of cluster lifecycle, certificate renewal, control plane upgrades, worker node upgrades, installing addons, and more, to be driven declaratively: you describe desired state in a CRD, and a controller reconciles it, the same model Kubernetes already uses for everything else it manages. That's also the direction that leads toward adopting Cluster API (CAPI) as the machinery behind that lifecycle management, rather than Heir maintaining its own bespoke imperative path alongside a growing declarative one.

This release's slice of that shift: previously, standing up a Heir cluster meant running the CLI directly against a management cluster. As of this release, the recommended path is to install the Operator chart and let it reconcile Runtime and WorkerJoinToken resources you apply with kubectl:

helm install heir oci://ghcr.io/tardigradeproj/charts/heir --version 1.35.7-heirv0.1.0 \
  -n heir-system --create-namespace \
  --set manager.image.repository=ghcr.io/tardigradeproj/heir-controller-manager \
  --set manager.image.tag=v1.35.7-heirv0.1.0

The chart covers the controller Deployment, RBAC helper roles, CRD install/keep behavior, the metrics endpoint, and optional cert-manager and Prometheus ServiceMonitor integration. See the updated Quickstart for the full walkthrough, and the Helm Chart reference for every value it accepts.

Plane Tunnel replaces Konnectivity​

Heir no longer depends on the Kubernetes Konnectivity proxy to reach worker nodes. Konnectivity assumes the API server is a fixed, singleton process: each konnectivity-agent holds a persistent tunnel to every konnectivity-server replica, so total connections grow as N×M, adding a replica forces every existing agent to reconnect before traffic can reach it, and --server-count has to be kept in sync with the actual replica count by hand. Heir runs the API server as a replicated, horizontally-scalable Deployment, an assumption Konnectivity was never built for.

The plane tunnel keeps the same N×M connection shape, one multiplexed TCP tunnel per worker per replica, but removes the coordination problems around it:

• Every agent dials every replica, not a routing table's worth. The agent resolves the plane tunnel's headless Service via DNS (--replica-discovery-dns), learns the current replica count, and keeps that many tunnels open, reconnecting as replicas come and go.
• No wrong-replica scenario. Because every replica already holds a tunnel to every worker, any replica can serve any request.
• Scaling out doesn't drop existing tunnels. A new replica just adds a DNS record; agents pick it up on their next reconnect and open one more tunnel without disturbing the others. There's no --server-count to keep in sync.

Each worker's tunnel is authenticated with mutual TLS, using the node's system:node:<name> client certificate, verified against the cluster CA, so the plane tunnel knows exactly which worker owns a tunnel with no separate identity protocol. Routing a request (kubectl exec, log streaming, or the API server's own reach into a kubelet) works the same way it does today with Konnectivity's egress selector: the API server issues an HTTP CONNECT worker2:10250 over its own mTLS leg to the plane tunnel Service, any replica accepts it, looks up the already-open agent tunnel for worker2, and dials the kubelet through it. The plane tunnel never terminates the inner TLS session between the API server and the kubelet, it just splices bytes between the two legs, so certificate verification on both ends behaves exactly as it would over a direct connection. Full design details, including the egress selector configuration, live in docs/plane-tunnel.md in the controller repo.

Why this exists beyond replacing Konnectivity​

Konnectivity only ever solved one problem: letting the API server reach a kubelet. The plane tunnel is built to solve that same class of problem for everything else Heir wants to run centrally, not just kube-apiserver. Every worker already holds one multiplexed connection into the management cluster; anything that needs to talk to something on that worker can register itself as another upstream on the same connection instead of asking for its own inbound port.

That's the actual reason it changed: it's the foundation for keeping Heir's "run control plane components centrally" philosophy consistent.

Multiplexing with outbound, not HTTP/2​

The plane tunnel's transport is outbound, a Go fork of HashiCorp's yamux that Heir maintains. outbound multiplexes streams at the transport layer, framing them directly over a raw TCP connection with a 13-byte header, instead of layering multiplexing inside HTTP/2's request/response semantics. That makes it cheaper than HTTP/2 for this job: no HTTP stack to run on either end, no request/response framing to pay for, just streams over a socket.

Closing a real allocation leak in the tunnel's copy path​

outbound's Tunnel proxies data between an agent stream and the real upstream connection with a small bidirectional pipe that used to call io.Copy(dst, src). net.TCPConn implements io.ReaderFrom/io.WriterTo, so io.Copy silently skipped its own pooled buffer and allocated a fresh scratch buffer per direction on every proxied connection. The old io.Copy path was using 2.4x more memory than the pooled version.

The fix, outbound commit 4522a0b, pools those buffers in a sync.Pool and hides net.TCPConn's fast-path interfaces so io.CopyBuffer is forced to actually use them:

var copyBufferPool = sync.Pool{
	New: func() any { buf := make([]byte, 32*1024); return &buf },
}

func tunnelPipe(a, b net.Conn) {
	pipe := func(dst, src net.Conn) {
		bufp := copyBufferPool.Get().(*[]byte)
		defer copyBufferPool.Put(bufp)
		_, _ = io.CopyBuffer(onlyWriter{dst}, onlyReader{src}, *bufp)
	}
}

go tool pprof -top mem.out confirms it: io.copyBuffer now shows up going through the pool, and the rest of the profile is ordinary Session/Stream/TLS bookkeeping, not a scratch buffer reallocated on every connection.

A declarative way to join workers​

heir token generate is now backed by a WorkerJoinToken custom resource. Apply one scoped to a Runtime, and the controller mints a one-shot bootstrap token and kubeconfig, publishes it as an owned <name>-jointoken Secret, revokes it on deletion, and re-issues it if the Secret is ever edited or deleted out from under it:

apiVersion: controlplane.tardigrade.runtime.io/v1alpha1
kind: WorkerJoinToken
metadata:
  name: new-node
  namespace: default
spec:
  runtimeRef:
    name: heir-quickstart
  ttl: 3h

JOIN_TOKEN=$(kubectl get secret new-node-jointoken -o jsonpath='{.data.jointoken}')
sudo heir provision worker --token="$JOIN_TOKEN"

A redesigned Runtime API​

The Runtime spec picked up a round of cleanup this release, all reflected in the Configuration Options reference and the Quickstart:

• upstreamCluster is now cluster
• The shared controlPlaneEndpoint.addresses list is gone. The API server and plane tunnel each get their own host under cluster.controlPlaneExternalEndpoint, since they can now sit behind different IPs
• ControlPlaneEndpointSpec is renamed ControlPlaneExternalEndpointSpec
• The plane tunnel's NodePort moved out of the shared service spec into spec.controlPlane.planeTunnel.service
• RegisterSetting is renamed RegistrySettings on the CoreDNS and kube-proxy specs
• ComponentEndpoint.Host is now validated with a CEL rule requiring a bare hostname or IPv4 address, since it's formatted directly into a cert SAN

This is a breaking change to the Runtime spec. If you're upgrading an existing cluster, update your manifests before applying this version's CRDs.

Sharper certificate and status visibility​

Runtime.status reports per-certificate expiry now, one entry per PKI cert (ca, apiserver, serviceaccount, planetunnel, apiserver-planetunnel, plus the scheduler and controller-manager client certs) instead of a single certificatesExpireAt field, and carries observedGeneration so you can tell whether status reflects the latest spec. WorkerJoinToken also gained printer columns, so kubectl get workerjointoken gives you the owning Runtime, readiness, reason, and expiry at a glance.

Under the hood​

• The heir-controller-manager image is now built by goreleaser, the same pipeline that builds heir and heir-tunnel, instead of a one-off Dockerfile
• Dependency versions (Kubernetes, containerd, runc, crictl, CNI, Calico) are centralized in .dependencies-version.yaml; this release bumps Kubernetes to v1.35.7 and crictl to v1.35.0 to match
• goreleaser now supports -rc prerelease tags, which is how v1.35.7-heirv0.1.0-rc1 shipped ahead of this stable cut
• End-to-end tests build real binaries and provision an actual Runtime and worker join, run in CI via shared, de-duplicated build-toolchain GitHub Actions
• docs/running-locally.md was rewritten as a full local-dev walkthrough and linked from the README and contributing guide

Upgrading​

Because the Runtime API changed shape, treat this as a breaking release: re-apply CRDs, update any Runtime manifests using the old field names, and switch worker provisioning over to WorkerJoinToken. The Quickstart and Configuration Options docs are fully updated for v1.35.7-heirv0.1.0 and are the fastest way to see the new shape end-to-end.

If you try to do an exercise of understanding the details of each piece that makes reading this article possible, you would need to rediscover the whole world, and it would take you more than 1,000 lifetimes. This means one thing: we don't understand 99.999% of the things that we interact with on a daily basis, but still, we trust them with our lives. Some are so well-designed that there are ones who may think that they are features that the world came with from the start. And there's nothing wrong with that, quite the opposite, that's how communities are supposed to function.

The interesting point is that these technologies didn't appear out of nowhere; they started from nothing, from imprecise beginnings, and over time became reliable because groups of people worked and designed them in a way that made them trustworthy. And if there hadn't been other groups of people to pick up where the last group left off, the technology would have died, and we would have collapsed as a society.

This is what we want to celebrate with Heir: a world in which you don't need to know everything, a society where each generation doesn't need to rediscover the world, a world of heritage.

Heir is not the last Kubernetes distribution. Kubernetes is not the last container orchestration tool. Containers are not the last Linux namespace abstraction. Linux is not the last operating system. x86 and ARM are not the last CPU architectures. Transistors are not the last switching mechanism. Silicon is not the last material. Nothing is good enough to be considered final, everything is just a bridge to what comes next, something bigger and better.