{"id":29305,"date":"2026-09-12T10:02:01","date_gmt":"2026-09-12T08:02:01","guid":{"rendered":"https:\/\/blog.mi.hdm-stuttgart.de\/?p=29305"},"modified":"2026-09-12T10:12:12","modified_gmt":"2026-09-12T08:12:12","slug":"leased-cloud-machines-teardown-guarantee","status":"publish","type":"post","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2026\/09\/12\/leased-cloud-machines-teardown-guarantee\/","title":{"rendered":"Leased cloud machines: making a teardown guarantee observable"},"content":{"rendered":"\n

Written as the course project for Software Development for Cloud Computing (113479a) at HdM Stuttgart. The source code for horizon<\/a> and bedrock<\/a> is on GitHub.<\/em><\/p>\n\n\n\n

\n

“This raises the question of who notices when the alarm system is not working properly.”<\/p>\n\n\n\n

Lisanne Bainbridge, Ironies of Automation<\/a>, 1983<\/p>\n<\/blockquote>\n\n\n\n

A machine you have to give back<\/h2>\n\n\n\n

Some jobs want a machine I don’t own. Not often, and never for long. An hour of eight cores for a build that would otherwise run overnight, on hardware I would happily own and can’t justify buying right now, with RAM and storage priced the way they are.<\/p>\n\n\n\n

Renting one is a single API call. The machine appears in about a minute and charges a few cents an hour from the moment it exists. Giving it back reliably took four months.<\/p>\n\n\n\n

That sounds like the wrong way round, so here’s the part that makes it true. A server bills for as long as it exists, whether or not anything is running on it, and nothing in the cluster notices when one is left behind. A cluster with a forgotten machine in it looks exactly like a cluster without one. Both report healthy. The difference turns up on an invoice, about a hundred euros later.<\/p>\n\n\n\n

horizon<\/a> is what I built for that. It’s a Kubernetes operator<\/a>: you write down how much capacity you want and until when. Then it rents the machines, joins them to the cluster, and destroys them when the time is up. It runs on bedrock<\/a>, which is three small Lenovo desktops at home running NixOS<\/a> and K3s<\/a>, reconciled from a git repository by Flux<\/a>.<\/p>\n\n\n\n

So the whole project is one requirement with a lot of machinery under it. When the deadline passes, the machine is gone.<\/p>\n\n\n\n

It’s easy to state, easy to implement badly, and surprisingly hard to know you have. The hard part was never deleting a machine. It was knowing, at any given moment, that it would be deleted. I thought I knew more than once, and I was wrong every time.<\/p>\n\n\n\n

What was pitched, and what survived<\/h2>\n\n\n\n

The original proposal was one sentence, and I’ve still got it:<\/p>\n\n\n\n

\n

a Go CLI and lightweight controller that lets any K3s homelab extend into the cloud, temporarily bursting overflow workloads, running permanent cloud workers, or federating independent clusters, using Terraform, NixOS, and Tailscale.<\/p>\n<\/blockquote>\n\n\n\n

Three products in one sentence. Bursting, permanent cloud workers, and cluster federation aren’t one tool with three settings; they are three tools that happen to rent machines. Bursting is about deadlines and teardown. Permanent workers are a different job made of upgrades, drain policy and drift over months. Federation is a third job again, whose hard parts are identity, trust between control planes, and what happens when the link goes down. I built parts of all three, and they kept getting in each other’s way.<\/p>\n\n\n\n

Only bursting survived, so horizon adds temporary capacity to a cluster that already exists.<\/p>\n\n\n\n

The named technologies fared no better. Terraform<\/a> was the first version’s entire provisioning layer. horizon shelled out to a module living in the bedrock repository, ran init<\/code> against it, and selected a workspace per burst before applying and reading the server outputs back. NixOS is still how bedrock’s machines are declared, and horizon knows nothing about it; the tool generates a cloud-init<\/a> document and boots whatever image the operator points it at. Tailscale<\/a>, the third name in the sentence, is configured entirely in the other repository.<\/p>\n\n\n\n

Most of the original scope ended up deleted. A Cluster API integration, a Velero-based workload migration and a full terminal interface were each built, used for a while, and taken out again.<\/p>\n\n\n\n

\"Six<\/a><\/figure>\n\n\n\n

That sounds like a waste of time, and yes, some of it was. But the deletions weren’t random, because most of what I cut turned out to belong somewhere else. Cluster creation belongs in the git tree that Flux reconciles, image building is the job of whoever owns the image, and the network overlay is the estate’s problem. What was left after every one of those cuts was the same requirement that had been in the sentence from the very beginning, buried in the middle of it: give the machine back.<\/p>\n\n\n\n

Everything after this point is an attempt to make that requirement true, and then an attempt to find out whether it actually is.<\/p>\n\n\n\n

Why the provider won’t do this for you<\/h2>\n\n\n\n

The obvious thing to do with an idle machine is switch it off. It buys nothing. A Hetzner server bills for as long as the server object exists, because a powered-off server keeps its capacity reserved; Hetzner’s billing documentation<\/a> is explicit that it keeps costing, and that any fraction of an hour is billed as a whole hour. Deletion is the only thing that stops the meter.<\/p>\n\n\n\n

Providers differ on whether a machine can arrange its own deletion, and they differ in a way that changes the architecture rather than the configuration.<\/p>\n\n\n\n

AWS will terminate an instance when the guest shuts itself down, if InstanceInitiatedShutdownBehavior<\/code><\/a> is set to terminate. Billing ends<\/a> when the state reaches shutting-down. The machine needs no credential at all; poweroff<\/code> is the whole mechanism. Google Compute Engine goes further and enforces a server-side maxRunDuration<\/code><\/a>, which can delete the instance when the timer expires. The provider holds that timer, where nothing on the machine can reach it.<\/p>\n\n\n\n

Hetzner has neither. There is no server-side deadline, and shutting down stops nothing. If a Hetzner machine is going to destroy itself, it has to call the delete API, and to call the delete API it has to hold a token that can delete servers. That token has to live on the ephemeral machine, which is also the machine running migrated workloads, and Hetzner tokens are project-scoped<\/a> with no per-resource permissions.<\/p>\n\n\n\n

I picked Hetzner for three reasons. It’s cheap. Cheap enough that another project written up on this blog<\/a> moved a startup’s backend off Heroku and onto Hetzner servers, and worked out that comparable capacity on Heroku would have cost five to six times as much. It is the hardest of the three, so building against it forces the compensating machinery to be real. And I wanted the mechanism to be mine and to work on any provider, instead of leaning on a feature that only a bigger provider offers and being tied to that provider to keep it. A design that works on Google Compute Engine may only be working because Google is holding the timer.<\/p>\n\n\n\n

The provider differences matter less than what they all have in common, which is that the failure mode is money, and money is silent. A crashed process pages someone, but a machine nobody deleted just keeps billing until whoever reads the invoice weeks later asks what that line is.<\/p>\n\n\n\n

First answer: the guarantee lived inside the program<\/h2>\n\n\n\n

At first the guarantee lived inside a single function, and for a while I couldn’t see anything wrong with that.<\/p>\n\n\n\n

core.Burst<\/code> did the whole job linearly. It reserved capacity from the provider and waited for the new node to register with the cluster, then rewrote the target workload’s affinity<\/a> so the pods would land on it, and returned. Two deferred<\/a> rollbacks guarded it: an outer one that returned the reserved server count to its previous value if anything after the scale step failed, and an inner one that restored the saved affinity and tolerations<\/a> if migration failed. Both ran on a fresh context, so a cancelled burst still cleaned up after itself.<\/p>\n\n\n\n

I read it twice, looking for the bug, and didn’t find one. Within its own assumptions it is correct.<\/p>\n\n\n\n

So I stopped reading and tried to break it instead. The experiment ran against the real core.Burst<\/code> and the real node image, one rented server per run, split into a treatment arm, where the process got killed, and a control arm, where it didn’t. In the treatment arm I sent the process SIGKILL<\/code><\/a> while it was waiting for the node to register: the kill a program cannot catch, refuse or clean up after, which is what a hard power-off or an out-of-memory kill looks like from inside the process. Five runs, five servers left running and billing, with nothing anywhere that knew they existed. In the control arm I let the node-ready timeout expire normally, so the function returned through its own error path. That run leaked nothing: the rollback fired and put the reserved count back where it had been.<\/p>\n\n\n\n

The control arm is what makes the result mean anything. Without it, five leaks look like five bugs in the rollback, and the fix looks like reading the rollback more carefully. With it, the rollback is demonstrably fine. It handles the failure it was written for, and it is absent for the failure it was not, because a deferred call runs when a function returns and a killed process never returns from anything.<\/p>\n\n\n\n

That is true in Go, in C, and in any other language, because the handling happens in the kernel and the process is never consulted. There is no amount of care inside a linear function that fixes this, and no signal handler to write. The cleanup is in the wrong place.<\/p>\n\n\n\n

The replacement<\/h3>\n\n\n\n

The replacement is a custom resource<\/a> and a controller. A CapacityLease<\/code> states how much capacity, from which provider and region, for how long, and for which workload. The controller compares that statement against observed reality on every pass and acts on the difference, so compensation stops being a deferred call and becomes the ordinary business of a loop that, by construction, runs again after a crash. Around it sit layers that do not share a failure mode: a finalizer<\/a> persisted before any provider call, ownership and deadline labels written atomically in the create call so an instance is never anonymous even for an instant, a collector that lists every configured provider on a slow tick and deletes anything no live lease claims once its deadline has passed, and launch deadlines that release a machine that never boots or never joins.<\/p>\n\n\n\n

I copied most of that shape from Karpenter<\/a>. I went through its published failure history expecting to find the bug I’d just written, and didn’t: the documented leaks there came from identity mismatch, not from missing cleanup code. That was a relief, and it was also a fairly direct instruction about where to look.<\/p>\n\n\n\n

Deletion counts as complete only when the provider reports the instance absent. A successful delete call is not evidence.<\/p>\n\n\n\n

How a machine in a datacentre joins a cluster in a house<\/h2>\n\n\n\n

Before any of that can be tested, a rented machine in Helsinki has to become a node in a cluster sitting on a residential line in Germany. These two things share no network, and the hard requirement is that nothing on the home line accepts an inbound connection.<\/p>\n\n\n\n

That requirement isn’t paranoia. Opening a port to a cluster’s API server from the public internet, on a connection whose address is already published in DNS, means the first line of defence for the whole estate is whatever the API server’s authentication does that day. I wasn’t going to open that port, but the rented machine still had to reach in.<\/p>\n\n\n\n

The way out is that both ends dial out. A coordination service, reachable from anywhere, learns where each peer currently is. The peers ask it about each other, then attempt to connect directly, punching through their respective network address translation<\/a> using the techniques described in Tailscale’s write-up of NAT traversal<\/a>. Once that succeeds the traffic is peer to peer and the coordination service is out of the path. Nothing points inward. The home firewall never gets an inbound rule, because there’s no inbound connection to permit: both sides opened outbound ones and met.<\/p>\n\n\n\n

\"A<\/a><\/figure>\n\n\n\n

There’s a second thing that had to happen first, which I hadn’t accounted for and which took over the project for a while.<\/p>\n\n\n\n

My cluster didn’t really have a network of its own. It was three machines on a flat home LAN, behind a consumer router from the telephone company. That router can’t express VLANs at all, and it can’t express a rule between two of its own devices. Everything I wanted to say about the boundary between a rented machine and my home could not be said on that hardware.<\/p>\n\n\n\n

So a Raspberry Pi became the router. It runs NixOS, declared as another host in the same repository as the cluster, with nftables<\/a> for the firewall, kea<\/a> for DHCP and AdGuard<\/a> for DNS. On top of that went a documented address plan. The cluster sits in a trusted servers zone of its own, the home devices are somewhere else entirely, and traffic between zones is denied by default. Changing the firewall is now an edit to a file and an apply, reviewed the same way as everything else.<\/p>\n\n\n\n

The alternative was an off-the-shelf firewall appliance, which does all of this behind a polished interface and keeps its configuration in its own XML on its own disk, outside the repository. It doesn’t run on the ARM hardware I already owned, so it meant buying another old PC. That is the same spend I could not justify at the top of this post. I picked the Pi and hand-assembled what the appliance bundles, which was more work and more networking than I had done before.<\/p>\n\n\n\n

Five overlay networks, one search<\/h2>\n\n\n\n

The join path above sounds like a design decision. In practice it was one requirement discovered slowly, at the cost of four wrong answers.<\/p>\n\n\n\n

The requirement is this. The coordination service is the thing every node has to reach before it can reach anything else. A rented machine that can’t reach it never learns where the cluster is, and a machine that never learns where the cluster is never becomes a node. So the coordination service has to be reachable from outside the house, unconditionally, before anything else works.<\/p>\n\n\n\n

I resisted that for a long time, because it meant giving up on hosting everything myself.<\/p>\n\n\n\n

The five attempts, in order<\/h3>\n\n\n\n

NetBird<\/a> went in first and lasted about five hours. It’s WireGuard with a management server and a signalling server on top, handing out keys and introducing peers to each other, and you can run both yourself. It cost a Helm<\/a> release, a dashboard behind the ingress, a management gRPC service on a node port and an encryption key in the secret store, and all of that came back out the same evening, when I replaced it with Headscale<\/a>.<\/p>\n\n\n\n

Everything in Tailscale is open except the control server, so Headscale is a rewrite of that one piece: the normal Tailscale clients point at your box instead of theirs. That one ran for a few days and failed on the requirement directly, rather than on any defect of its own. I had deployed it into the cluster, so the coordination server every node needed in order to find the cluster was itself inside the cluster, behind the residential line, where reaching it from outside meant opening exactly the inbound path I had ruled out. A rented machine booting in Helsinki couldn’t get its first packet anywhere.<\/p>\n\n\n\n

ZeroTier<\/a> came next, on a suggestion from my professor, and held for about six weeks, the longest of the four that lost. It doesn’t use WireGuard at all, and it isn’t really a VPN in the sense the others are. It’s a virtual Ethernet switch: joined machines behave as though they’re plugged into the same LAN, and they find each other through root servers ZeroTier runs. Joins are one command, so rented nodes came up on it. What it didn’t do properly was remote access, and there were things I could not route through its tunnels. That is what took it out.<\/p>\n\n\n\n

Then I went the other way entirely and self-hosted WireGuard<\/a> on the router. WireGuard is the protocol most of the others are built on, and it’s only the tunnel, which was the appeal: it’s in the kernel, it’s declarative, it answers only authenticated peers, and it puts nobody else in the path.<\/p>\n\n\n\n

That lasted under two days, and it failed on a design position rather than a bug. WireGuard does no peer discovery, deliberately: a peer is a public key and an endpoint that somebody typed in. That’s a virtue when the peer list is a fixed set of machines. It is fatal when the entire point of the system is that a machine which didn’t exist five minutes ago has to join by itself. Every rented node needed key material injected at provisioning time and a one-shot unit<\/a> to bring the interface up before K3s could start, and the home line had to publish a stable endpoint whose address couldn’t go in the repository.<\/p>\n\n\n\n

Tailscale is WireGuard plus a coordination server, which swaps public keys between machines and walks them through NAT so neither side needs an open inbound port. It solves discovery, and it solves it the one way I had spent weeks refusing: by letting a company run that server.<\/p>\n\n\n\n

The requirement that ruled out four of the five is the one I was slowest to accept, and accepting it cost me a position I had already defended in writing. Earlier I had turned down a small VPS on the grounds that it reintroduced somebody else’s computer into a setup meant to be self-hosted. Then I handed the same job to a company. Once somebody else’s computer was in the picture either way, a VPS running Headscale would only have added one more server for me to patch and keep online. It isn’t a flattering ending, but it bought something real. I’m not paying Tailscale anything, and if Cloudflare goes down again I can still reach my cluster from anywhere in the world.<\/p>\n\n\n\n

Second answer: the guarantee moved onto the machine<\/h2>\n\n\n\n

All four of the operator’s own layers have something in common, and it took me a while to see it. The finalizer, the ownership labels, the launch deadlines and the collector each close a different race. Every one of them needs the operator process, the API server and the network alive at the same moment. That combination is the one most likely to fail together, and it is the combination a leaked server usually implies. So a fifth cluster-side layer would have improved the ordinary case and done nothing at all for the case that matters.<\/p>\n\n\n\n

So the guarantee had to move off the cluster and onto the rented machine itself. What runs there is a dead man’s switch: it arms itself when the machine boots, and it destroys the machine when whichever of two deadlines comes first runs out.<\/p>\n\n\n\n

The first clock is a time of day. Kubernetes keeps a record for every machine in the cluster, and the controller writes the deadline onto that machine’s own record, as an annotation<\/a>, which is a small piece of text attached to it. The agent reads its own Node<\/code> using the kubelet<\/a> credential the machine already has, so it needs no account of its own. Kubernetes keeps that credential weak on purpose, and when I tried it, it couldn’t read any other node, couldn’t list nodes, and couldn’t even delete its own. The machine asks for its deadline, and the cluster never has to reach in and tell it. So nothing needs a way into a firewalled machine, and no new credential has to exist. A refreshed annotation can push this deadline later, but never past the second one.<\/p>\n\n\n\n

The second clock doesn’t care what time it is. It starts when the agent starts and counts forward from there, and nothing can extend it. It uses Go’s monotonic clock<\/a>, which a change to the system time can’t move. Setting the clock back, by accident or on purpose, buys a machine nothing.<\/p>\n\n\n\n

\"Five<\/a><\/figure>\n\n\n\n

The agent fires if either clock has run out. That is the whole decision, and the function that makes it has no third case:<\/p>\n\n\n\n

func fired(startedAt time.Time, maxLifetime time.Duration, wall *time.Time, now time.Time) (string, bool) {\n\tif now.Sub(startedAt) >= maxLifetime {\n\t\treturn reasonMaxLifetimeElapsed, true\n\t}\n\tif wall != nil && !now.Before(*wall) {\n\t\treturn reasonWallClockDeadline, true\n\t}\n\treturn "", false\n}<\/code><\/pre>\n\n\n\n
A third case would let one clock speak for the other, and neither can be trusted alone. The time of day can be defeated by editing the deadline or moving the clock. The counter can’t express a lease that gets extended, only a fixed maximum. More to the point, the controller writes the annotation, and the controller is precisely the component whose failure this layer exists to survive. A machine that trusts the annotation to tell it when not to fire is trusting a dead process.<\/p>\n\n\n\n
horizon treats a node that can’t reach the cluster as dead, on purpose. It isn’t running scheduled work and it’s pure cost, so the switch simply fires, whether the cluster is down or just unreachable. The deadline always sits a margin beyond the renewal interval, and the cluster refuses a provider configuration where that margin is too small to be useful. The margin is what a missed renewal is allowed to cost before the switch fires.<\/p>\n\n\n\n
When the switch fires, the delete keeps retrying: first after five seconds, then doubling, then once a minute for as long as the process is alive. The retry count is set to the largest 32-bit integer, which at that spacing is a few thousand years. Giving up would leave a server billing, and a server that is billing is the thing this code exists to prevent, so there is no failure branch worth taking.<\/p>\n\n\n\n
The turn: the second clock was not running<\/h2>\n\n\n\n
Either clock is enough on its own to destroy the machine, the one the controller keeps refreshing or the one the agent starts at boot.<\/p>\n\n\n\n
I’d tested that logic hard. It never occurred to me to test whether the second clock was running at all.<\/p>\n\n\n\n
A leased node came up, joined, and reported Ready. It took work. Its watchdog had never started.<\/p>\n\n\n\n
The cloud-init horizon writes is supposed to download the agent and install a unit to run it. But on NixOS that script runs with a PATH that misses the system profile. busybox<\/a> was supplying most of what the script reached for, and curl wasn’t there at all, so the download failed and the unit was never written. The rest of the script carried on as if it had worked.<\/p>\n\n\n\n
Nothing noticed.<\/p>\n\n\n\n
Neither Kubernetes nor horizon had an opinion. A node is Ready<\/a> when its kubelet tells the API server it’s healthy, which says nothing about whether the machine can still destroy itself, and the lease showed instances ready and a deadline in the future.<\/p>\n\n\n\n
So for a while the guarantee ran on one clock, and the clock it lost was the one that exists for when the control plane is gone.<\/p>\n\n\n\n
The fix in the image was one line of PATH, putting the real tools ahead of busybox on all four cloud-init stage units. The boot script had been looking up its tools in the wrong order. So most of what it asked for came back as a stripped-down stand-in instead of the real program, and it carried on regardless. It only failed where anyone could see it, at the one tool the stand-in has no version of.<\/p>\n\n\n\n
The more important fix was in horizon, where the agent annotates its own node once the watchdog is armed and the controller turns that into a condition on the lease. Now the cluster can answer the question directly. Before, the only way to find out was to take a machine away and watch.<\/p>\n\n\n\n
The same shape, the same week<\/h2>\n\n\n\n
That same week I hit the identical failure with different parts.<\/p>\n\n\n\n
A leased node came up Ready and schedulable. Pods scheduled onto it. Those pods could reach nothing at all: a probe pod on that node couldn’t resolve kubernetes.default.svc<\/code>, and couldn’t reach a pod on a home node, timing out after twelve seconds. From the other direction, a home worker pinging the leased node’s overlay address got a hundred percent packet loss.<\/p>\n\n\n\n
The cluster reported the node healthy the entire time.<\/p>\n\n\n\n
\"The<\/a><\/figure>\n\n\n\n
The router’s firewall forwards a narrow list of ports from the overlay into the trusted zone, including the Kubernetes API port. So the kubelet on the rented machine reached the API server, registered, and kept its lease renewed, which was all the Ready condition asked for. But the pod network is a different interface with a different address range, and its traffic hit a drop rule a few lines below the allow.<\/p>\n\n\n\n
Kubernetes checked the one path the firewall permitted, found it working, and reported what it found, without ever looking at the pod network.<\/p>\n\n\n\n
The fix wasn’t on the rented machine. The three home nodes had to join the overlay themselves, and the pod network had to move onto it fleet-wide, so that every node in the cluster advertises an address from the same range and any peer is directly reachable from any other. That reverses two earlier decisions in the repository: one that pinned the home pod-network MTU on the explicit assumption that the home nodes had no overlay interface to bind to, and one that had removed the last home node from the overlay and concluded that no home node was a member any more.<\/p>\n\n\n\n
The rollout had to run in two separate phases, every node onto the overlay first and then the pod network node by node, control plane last, because the intermediate state is the actual hazard. A migrated node advertises an overlay address while an unmigrated peer still advertises a LAN address, and the two directions never agree on a route. That asymmetry had already produced a pod network passing zero traffic between two nodes, down to the smallest packets. That earlier incident is the reason the phases are separate at all. Stopping halfway makes it permanent.<\/p>\n\n\n\n
I didn’t trust the new setup until I’d checked by hand that a pod in Helsinki could reach the API server and talk directly to a pod at home.<\/p>\n\n\n\n
The price is that two machines on the same switch, a metre apart, now encrypt their pod traffic and send it through a tunnel to reach each other. Flannel<\/a> binds one interface per node, so there’s no configuration in which home-to-home traffic takes the LAN and home-to-rented traffic takes the tunnel. I measured the direct path between two home nodes at under a millisecond, so the overhead is small. But it’s still overhead paid on every packet for a machine that exists a few times a month.<\/p>\n\n\n\n
Third answer: the guarantee reports itself<\/h2>\n\n\n\n
Both failures had the same shape, and so did the fix. A guarantee that cannot be checked is a guarantee taken on faith, and faith is exactly what got proved wrong twice in a week.<\/p>\n\n\n\n
So the agent annotates its own Node<\/code> once its watchdog is armed, and refreshes that annotation on every poll tick that does not fire the switch. The controller reads the annotation for every joined instance and raises a WatchdogArmed<\/code> condition on the lease. A condition is Kubernetes’ standard way for an object to report a yes-or-no fact about itself. It’s true when every joined machine reports a fresh value, and false otherwise, with a warning naming the machines at fault. It shows up as a column in kubectl get capacityleases<\/code>, and because it’s an ordinary condition, kube-state-metrics<\/a> can turn it into a metric with a few lines of configuration. The cluster can alert on it without a line of code that knows anything about horizon.<\/p>\n\n\n\n
It’s an annotation rather than a label because the value is a timestamp, and label values can’t contain the colons a timestamp needs. A machine is already allowed to update its own record, under the node authorizer<\/a>, and that’s the same permission the deadline read depends on.<\/p>\n\n\n\n
The condition is purely a report, so it doesn’t gate scheduling, block a lease, or trigger teardown. Losing the watchdog is a loss of the layer that survives the control plane being unreachable, which is a loss of defence in depth. It is not a loss of teardown, and wiring the report into the enforcement would create a dependency between two things whose independence is the entire design.<\/p>\n\n\n\n
Because the agent keeps refreshing the annotation, a unit that keeps crashing, a script that never started, and an install that died before writing anything all look the same to the controller, which sees no fresh annotation and sets the condition to false.<\/p>\n\n\n\n
Measuring it<\/h2>\n\n\n\n
To find out whether any of it actually worked, I rented dozens of machines and drove them from a harness outside the cluster. It had to be outside, because the most interesting measurements meant killing the operator, and once the operator is dead, neither it nor Prometheus<\/a> can record anything about it. The harness polls the provider API directly every five seconds and records the exact instant a server disappears, which is the authoritative clock for teardown because it observes the disappearance instead of inferring it from a controller that may be dead. The harness reports everything the operator says about itself against that.<\/p>\n\n\n\n
Every machine in those runs was released and independently verified gone at the provider. Every run ends with a row written only after the harness queries the provider and sees no server carrying that run’s label. No run needed manual intervention, including six that failed for unrelated reasons and five in which the operator was deliberately scaled to zero.<\/p>\n\n\n\n
\"The<\/a><\/figure>\n\n\n\n
Where the controller had a working path, the operator’s recorded release time equalled the lease deadline to the second in every one of twenty-five runs, and the provider reported the server absent between zero and five seconds later, median three. That gap is at or below the harness’s own poll interval in every run, so the split between “the controller noticed” and “the machine went away” is not resolvable by this instrument. I’d designed the runs to measure that split. The honest answer is that it is smaller than I can see.<\/p>\n\n\n\n
The result I care about is the arm where the control plane was removed. Three runs took a twenty-minute lease, waited for readiness, then scaled the operator to zero replicas. With the operator gone the wall-clock deadline stopped being refreshed, so the last value it had written was the one that stood. In all three runs the machine deleted itself five, thirteen and nine seconds after that deadline, and roughly fifteen minutes before the lease deadline it was supposedly running to. The harness’s own log confirms the operator came back only after the server was already gone, so nothing cluster-side can account for the deletion.<\/p>\n\n\n\n
No control plane took part in those three deletions. The machines destroyed themselves from a clock they were carrying.<\/p>\n\n\n\n
This doesn’t prove the system can run without an operator, since both the deadline the node fired on and the token it deleted with were placed on the machine by the operator before it died. It only shows that teardown survives the operator going away.<\/p>\n\n\n\n
The instruments misled too<\/h2>\n\n\n\n
My own tests fooled me twice more. Thirty boots split cleanly into two groups by how long they took to become ready. One group of values sat just above one minute, the other just above ninety seconds, and there was an empty stretch in the middle with nothing in it at all. Almost every value also landed within a second above a multiple of thirty seconds. A physical process that produces two groups with nothing between them is interesting, and I spent a while being interested.<\/p>\n\n\n\n
The controller’s requeue interval, while those runs happened, was thirty seconds. It had no watch on Node<\/code> objects, so node readiness couldn’t reach the lease’s status between ticks. The split was the sampling grid.<\/p>\n\n\n\n
\"Thirty<\/a><\/figure>\n\n\n\n
The provider-side timings from the same runs do not land on that grid at all, which is what settles it: create latency is tight and forms a single group, and the gap from running to recorded ready is thirty-four to forty-five seconds in the fast group and sixty-two to sixty-nine in the slow one, with nothing between. So what I can defend is a range and not a number. The controller now wakes on Node<\/code> events and takes readiness from the node’s own timestamp, which fixes the instrument.<\/p>\n\n\n\n
Concurrent boots<\/h3>\n\n\n\n
The second one cost machines.<\/p>\n\n\n\n
I’d scheduled several measurement arms to run at once, on the correct reasoning that a provider which bills a whole hour per machine charges nothing extra for concurrency. What I hadn’t verified was that concurrency worked at all, because every boot until that point had been strictly sequential, which I confirmed afterwards by checking that no two runs overlapped.<\/p>\n\n\n\n
Under concurrency, joins became unreliable in two different-looking ways. Some machines sat at created and never joined, while others joined and reached Ready, only for the earlier of the two to lose readiness half a minute later while the later one carried on.<\/p>\n\n\n\n
Two deliberate concurrent boots and an SSH session gave the answer. Both machines reported the same overlay device identity, the same overlay address, and the same \/etc\/machine-id<\/code><\/a>. They weren’t two devices sharing an address, they were one device. The node image, a snapshot built from the bedrock repository, had baked a machine identity into itself, so every clone booted from it presented the same identity to the overlay’s coordination service. The later registration took the identity, and the earlier machine lost its route, which explains both signatures at once: lose the route before joining and the node never joins, lose it after joining and the node goes NotReady.<\/p>\n\n\n\n
The image had booted dozens of times without a hint of the problem. In sequential tests each machine was gone before the next one took the same identity, so there was never anything to conflict with.<\/p>\n\n\n\n
The fix is in bedrock’s image build and not in horizon: one provisioner that clears the machine identity and the overlay’s saved state immediately before the snapshot is taken, and then asserts both are cleared, so a build that fails to clear identity fails the build. The absence of that assertion is what let the defect hide for so long. I verified the fix against the exact scenario that had failed.<\/p>\n\n\n\n
What still cannot be seen<\/h2>\n\n\n\n
The guarantee survives the failure of either clock, but not the failure of both. With the operator scaled to zero and the node agent stopped, two machines outlived their lease deadlines by thirteen minutes and were collected five and six seconds after I scaled the operator back up. The sweeper that collects an abandoned machine lives inside the operator, so removing the operator removes the sweeper. That is a real hole, and it’s where the layering stops.<\/p>\n\n\n\n
Workload migration ships too, and naming one or more namespaces moves their workloads onto the leased nodes and restores them at teardown, with the saved placement written onto the workloads themselves so a restore survives the operator being removed entirely. There is also a replication mode that leaves the originals where they are and runs extra copies on the leased nodes until the lease ends, which works for Deployments but not StatefulSets, though neither mode was part of the measurement campaign.<\/p>\n\n\n\n
So far every machine the finished version rented has been deleted within the limits I set before the first run, all fifty-five of them. That doesn’t promise anything about the next one, but it’s the best answer I have.<\/p>\n\n\n\n
If I keep one thing from all of this, it’s that every failure here was a system telling the truth about something next to what I wanted to know. The deferred rollback handled the errors it was written for, the Ready condition described the API path, the lease reported a deadline for a machine that couldn’t enforce it, the sampling grid reported what it sampled, and dozens of sequential boots reported a working image. In each case the fix was to add a signal that checks the thing itself, instead of inferring it from something next to it.<\/p>\n\n\n\n
\n
“If some host holding a lease for this file is unreachable, the delay continues until the lease expires.”<\/p>\n\n\n\n
Cary G. Gray and David R. Cheriton, Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency<\/a>, 1989<\/p>\n<\/blockquote>\n\n\n\n
References<\/h2>\n\n\n\n
[1] L. Bainbridge, “Ironies of automation,” Automatica, vol. 19, no. 6, pp. 775-779, 1983. [Online]. Available: https:\/\/ckrybus.com\/static\/papers\/Bainbridge_1983_Automatica.pdf<\/a><\/p>\n\n\n\n
[2] The Kubernetes Authors, “Operator pattern,” Kubernetes Documentation. [Online]. Available: https:\/\/kubernetes.io\/docs\/concepts\/extend-kubernetes\/operator\/<\/a><\/p>\n\n\n\n
[3] NixOS contributors, “Nix & NixOS | Declarative builds and deployments.” [Online]. Available: https:\/\/nixos.org\/<\/a><\/p>\n\n\n\n
[4] The K3s Project Authors, “K3s – Lightweight Kubernetes.” [Online]. Available: https:\/\/docs.k3s.io\/<\/a><\/p>\n\n\n\n
[5] The Flux authors, “Flux – the GitOps family of projects.” [Online]. Available: https:\/\/fluxcd.io\/<\/a><\/p>\n\n\n\n
[6] HashiCorp, “Terraform,” HashiCorp Developer. [Online]. Available: https:\/\/developer.hashicorp.com\/terraform<\/a><\/p>\n\n\n\n
[7] Canonical Group Ltd, “Cloud-init documentation.” [Online]. Available: https:\/\/docs.cloud-init.io\/<\/a><\/p>\n\n\n\n
[8] Tailscale, “What is Tailscale?” Tailscale Docs. [Online]. Available: https:\/\/tailscale.com\/docs\/concepts\/what-is-tailscale<\/a><\/p>\n\n\n\n
[9] Hetzner Online GmbH, “FAQ,” Hetzner Docs, Cloud billing. [Online]. Available: https:\/\/docs.hetzner.com\/cloud\/billing\/faq\/<\/a><\/p>\n\n\n\n
[10] Amazon Web Services, “Change instance initiated shutdown behavior,” Amazon Elastic Compute Cloud User Guide. [Online]. Available: https:\/\/docs.aws.amazon.com\/AWSEC2\/latest\/UserGuide\/Using_ChangingInstanceInitiatedShutdownBehavior.html<\/a><\/p>\n\n\n\n
[11] Amazon Web Services, “Terminate Amazon EC2 instances,” Amazon Elastic Compute Cloud User Guide. [Online]. Available: https:\/\/docs.aws.amazon.com\/AWSEC2\/latest\/UserGuide\/terminating-instances.html<\/a><\/p>\n\n\n\n
[12] Google, “Limit the run time of a VM,” Compute Engine, Google Cloud Documentation. [Online]. Available: https:\/\/cloud.google.com\/compute\/docs\/instances\/limit-vm-runtime<\/a><\/p>\n\n\n\n
[13] Hetzner Online GmbH, “Generating an API token,” Hetzner Docs, Cloud API. [Online]. Available: https:\/\/docs.hetzner.com\/cloud\/api\/getting-started\/generating-api-token\/<\/a><\/p>\n\n\n\n
[14] The Kubernetes Authors, “Assigning Pods to Nodes,” Kubernetes Documentation, sec. Affinity and anti-affinity. [Online]. Available: https:\/\/kubernetes.io\/docs\/concepts\/scheduling-eviction\/assign-pod-node\/#affinity-and-anti-affinity<\/a><\/p>\n\n\n\n
[15] The Go Authors, “Defer statements,” The Go Programming Language Specification, version go1.27, 2026. [Online]. Available: https:\/\/go.dev\/ref\/spec<\/a><\/p>\n\n\n\n
[16] The Kubernetes Authors, “Taints and Tolerations,” Kubernetes Documentation. [Online]. Available: https:\/\/kubernetes.io\/docs\/concepts\/scheduling-eviction\/taint-and-toleration\/<\/a><\/p>\n\n\n\n
[17] Linux man-pages project, “signal(7): overview of signals,” release 6.19, 2026. [Online]. Available: https:\/\/man7.org\/linux\/man-pages\/man7\/signal.7.html<\/a><\/p>\n\n\n\n
[18] The Kubernetes Authors, “Custom Resources,” Kubernetes Documentation. [Online]. Available: https:\/\/kubernetes.io\/docs\/concepts\/extend-kubernetes\/api-extension\/custom-resources\/<\/a><\/p>\n\n\n\n
[19] The Kubernetes Authors, “Finalizers,” Kubernetes Documentation. [Online]. Available: https:\/\/kubernetes.io\/docs\/concepts\/overview\/working-with-objects\/finalizers\/<\/a><\/p>\n\n\n\n
[20] Amazon Web Services, “Karpenter: Just-in-time Nodes for Any Kubernetes Cluster.” [Online]. Available: https:\/\/karpenter.sh\/<\/a><\/p>\n\n\n\n
[21] P. Srisuresh and K. Egevang, “Traditional IP network address translator (Traditional NAT),” RFC 3022, IETF, Jan. 2001. [Online]. Available: https:\/\/www.rfc-editor.org\/rfc\/rfc3022.html<\/a><\/p>\n\n\n\n
[22] D. Anderson, “How NAT traversal works,” Tailscale, 2020. [Online]. Available: https:\/\/tailscale.com\/blog\/how-nat-traversal-works<\/a><\/p>\n\n\n\n
[23] The netfilter project, “The netfilter.org ‘nftables’ project.” [Online]. Available: https:\/\/netfilter.org\/projects\/nftables\/index.html<\/a><\/p>\n\n\n\n
[24] Internet Systems Consortium, “Kea Administrator Reference Manual,” Kea 3.2.0. [Online]. Available: https:\/\/kea.readthedocs.io\/en\/stable\/<\/a><\/p>\n\n\n\n
[25] AdGuard, “AdGuard Home: Network-wide ads & trackers blocking DNS server.” [Online]. Available: https:\/\/github.com\/AdguardTeam\/AdGuardHome<\/a><\/p>\n\n\n\n
[26] NetBird, “NetBird – Open Source Zero Trust Networking.” [Online]. Available: https:\/\/netbird.io\/<\/a><\/p>\n\n\n\n
[27] The Helm Authors, “Helm: The package manager for Kubernetes.” [Online]. Available: https:\/\/helm.sh\/<\/a><\/p>\n\n\n\n
[28] J. Font, “headscale: An open source, self-hosted implementation of the Tailscale control server.” [Online]. Available: https:\/\/github.com\/juanfont\/headscale<\/a><\/p>\n\n\n\n
[29] ZeroTier, Inc., “ZeroTier | Next-Generation Connectivity and Cybersecurity.” [Online]. Available: https:\/\/www.zerotier.com\/<\/a><\/p>\n\n\n\n
[30] J. A. Donenfeld, “WireGuard: fast, modern, secure VPN tunnel.” [Online]. Available: https:\/\/www.wireguard.com\/<\/a><\/p>\n\n\n\n
[31] The systemd project, “systemd.service(5): Service unit configuration,” systemd 261. [Online]. Available: https:\/\/man7.org\/linux\/man-pages\/man5\/systemd.service.5.html<\/a><\/p>\n\n\n\n
[32] The Kubernetes Authors, “Annotations,” Kubernetes Documentation. [Online]. Available: https:\/\/kubernetes.io\/docs\/concepts\/overview\/working-with-objects\/annotations\/<\/a><\/p>\n\n\n\n
[33] The Kubernetes Authors, “Cluster Architecture,” Kubernetes Documentation, sec. kubelet. [Online]. Available: https:\/\/kubernetes.io\/docs\/concepts\/architecture\/#kubelet<\/a><\/p>\n\n\n\n
[34] The Go Authors, “time package,” Go Packages, sec. Monotonic Clocks. [Online]. Available: https:\/\/pkg.go.dev\/time#hdr-Monotonic_Clocks<\/a><\/p>\n\n\n\n
[35] The BusyBox project, “BusyBox: The Swiss Army Knife of Embedded Linux.” [Online]. Available: https:\/\/www.busybox.net\/about.html<\/a><\/p>\n\n\n\n
[36] The Kubernetes Authors, “Node Status,” Kubernetes Documentation, sec. Conditions. [Online]. Available: https:\/\/kubernetes.io\/docs\/reference\/node\/node-status\/#condition<\/a><\/p>\n\n\n\n
[37] The flannel authors, “flannel: flannel is a network fabric for containers, designed for Kubernetes.” [Online]. Available: https:\/\/github.com\/flannel-io\/flannel<\/a><\/p>\n\n\n\n
[38] The Kubernetes Authors, “kube-state-metrics: Add-on agent to generate and expose cluster-level metrics.” [Online]. Available: https:\/\/github.com\/kubernetes\/kube-state-metrics<\/a><\/p>\n\n\n\n
[39] The Kubernetes Authors, “Using Node Authorization,” Kubernetes Documentation. [Online]. Available: https:\/\/kubernetes.io\/docs\/reference\/access-authn-authz\/node\/<\/a><\/p>\n\n\n\n
[40] The Prometheus Authors, “Prometheus – Monitoring system & time series database.” [Online]. Available: https:\/\/prometheus.io\/<\/a><\/p>\n\n\n\n
[41] The systemd project, “machine-id(5): Local machine ID configuration file,” systemd 261. [Online]. Available: https:\/\/man7.org\/linux\/man-pages\/man5\/machine-id.5.html<\/a><\/p>\n\n\n\n
[42] C. G. Gray and D. R. Cheriton, “Leases: An efficient fault-tolerant mechanism for distributed file cache consistency,” in Proc. 12th ACM Symp. Operating Systems Principles (SOSP), 1989, pp. 202-210. [Online]. Available: https:\/\/www.cs.cmu.edu\/~15712\/papers\/gray89.pdf<\/a><\/p>\n\n\n\n
[43] L. Walz, “bedrock: Bare-metal Kubernetes homelab on NixOS, reconciled with Flux GitOps,” 2025. [Online]. Available: https:\/\/github.com\/lucawalz\/bedrock<\/a><\/p>\n\n\n\n
[44] L. Walz, “horizon: Kubernetes operator for job-shaped burst capacity on Hetzner Cloud, torn down by construction,” 2026. [Online]. Available: https:\/\/github.com\/lucawalz\/horizon<\/a><\/p>\n\n\n\n
[45] E. Ngo, N. Brocker, B. Reuter, and M. Koch, “Migrating from Heroku to Hetzner: Achieving scalability with Docker, Kubernetes and Rancher,” Computer Science Blog @ HdM Stuttgart, Feb. 2021. [Online]. Available: https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2021\/02\/27\/migrating-from-heroku-to-hetzner-achieving-scalability-with-docker-kubernetes-and-rancher\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
Renting a cloud machine is a single API call. Giving it back reliably took four months. This post follows horizon through three rewrites, the teardown layers that survived them, and the measurements that show whether the guarantee actually holds.<\/p>\n","protected":false},"author":1238,"featured_media":29317,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false,"_ppma_block_editor_authors":""},"categories":[120,659,22,21],"tags":[7,332,421,338,154,1267,1266],"ppma_author":[1065],"class_list":["post-29305","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cloud-technologies","category-devops","category-student-projects","category-system-architecture","tag-cloud","tag-go","tag-hetzner","tag-k3s","tag-kubernetes","tag-nixos","tag-tailscale"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t