A minimal GitOps reconciler for a simulated greenhouse edge device.
Live Demo
Introduction
As software developers we’re kind of trained to expect that anything that can go wrong eventually will, so we build fallbacks, write tests, and generally design around failure.
I spend a lot of my time automating devices that are annoyingly far from a keyboard. I containerized 3D printers with a print-management layer at the VS lab, I’ve built digital twin software for live manufacturing or DevOps workflows at my job.
These things already taught me that if something can go wrong it mostly will. To start off, I want to share a personal story that changed how I’m thinking during the architectural phase of a project.
Some things can’t be predicted
Recently I started to build a sound reactive stage light setup. A friend of mine was hosting an event in Belgium, and the relevant detail is that she has an entire abandoned castle in her backyard. I also happen to know someone who makes electronic goth music, which fits an abandoned castle quite well, so I packed my lights and my friend into a car and drove out to rural Belgium to play a concert with her and the lights.
The lights are controlled over multiple ESP32 Microcontroller by talking to my main machine over a switch. At that point I basically had a small IoT system running the show. Everything worked fine when the concert started.
And then the ceiling comes down onto one of the sculptures in the room. The reasonable reaction here is to leave the building, but this is a post about managing edge devices and not about building safety standards, so we’ll move on.
It didn’t destroy the lights, but somewhere in the chaos one of the controllers crashed. Because I was also running the backing audio for the concert at the same time I couldn’t just walk over and reboot it. The stage stayed dark for about ten minutes. The fix was maybe five meters away and I still couldn’t get to it. During that experience it would have been nice if the lights were somehow able to repair themselves.
The bigger problem
But no, I didn’t build a cloud native self-healing stage light system after that. I didn’t reach that level of insanity yet. But it reminded me of this broader problems a lot of systems are facing. Let’s scale it up to somewhere the fix genuinely isn’t five meters away. Say you have 10,000 greenhouses spread across remote Australia and the nearest technician is 500 kilometers away. If one drifts out of its safe range and nobody notices in time, the plants just cook. The thing that’s broken is too far away for a person to reach, so the fix has to come from the system itself.
A few years back, one approach to exactly that got a name, GitOps, and that’s what the rest of this post is about. My light controller rebooting itself would’ve counted as self-healing. GitOps adds the other half, it doesn’t just come back on, it comes back into the exact configuration you actually declared, every time. That’s what my course project builds. A small program that reads a desired configuration from a git repo and continuously keeps a running device matching it, for a simulated greenhouse.
What GitOps actually is
GitOps has two ingredients. A git repository holds the desired state of the system as data, not scripts that change the system, a plain description of what it should look like, so changing the system means changing that file and committing. And an automated process continuously makes the running system match that data, comparing declared state to reality in a loop and fixing whatever’s different, forever, not once.
The second ingredient is the one people usually miss. A CI/CD pipeline that deploys on push is a one-shot thing, it applies a change and then it’s done, and if something drifts an hour later nothing notices. GitOps keeps checking and keeps correcting, so the declared state stays enforced instead of applied once and forgotten. In this project the repository is hdm-sdcc-state, the data is one file called greenhouse.yaml, and the process is a small pod called the reconciler.
The loop itself does four things and then sleeps:
1. desired = read the declared state from git
2. observed = ask the device what state it is actually in
3. diff = the parts where observed does not match desired
4. if diff is not empty: issue the changes that close the gap
-> sleep 8 seconds
It keeps no memory of previous cycles. It doesn’t track what it did last time. It just looks at the world right now and asks one question, does this match the file. If the pod crashes mid-cycle, or the device was unreachable for a minute, the next successful cycle asks the same question about the same world and does the same fix, so there’s nothing to fall behind on. This is called level-triggered, as opposed to edge-triggered, which is “when X happens, do Y” and is efficient but fragile, because a missed event just never fires and nothing catches up. A missed level-triggered cycle costs basically nothing, since the next one re-evaluates from scratch. Kubernetes controllers are famously hard to break despite doing something conceptually simple, and this is a big part of why, they don’t remember what happened, they only look at now.
The two halves of a reconciler have names too. Desired state, which Kubernetes calls spec, is what should be true. Observed state, which Kubernetes calls status, is what’s currently true. A reconciler continuously drives observed toward desired. In this project the device’s API was deliberately shaped so both sides share the exact same structure, five named actuators, always all present, plain on-or-off strings, which turns the whole comparison into a five-key string compare with no conversion step. It’s spec and status semantics without a CRD, without etcd, and without touching the Kubernetes API at all.
What I built
A from-scratch GitOps reconciler driving a simulated greenhouse edge device, in four small pieces. The device is framework-free PHP, three files, simulating temperature, moisture, and light through five actuators(pump, fan, vent, lamp, door), serving a plain HTTP API and deliberately not saving its own state across a restart. The state repo is one YAML file, five actuator values, and editing and committing it is the entire control surface of the whole system. The reconciler is the loop above, clone the state repo, read the file, ask the device its state, correct the difference, sleep, repeat, in just under 340 lines of PHP across four files. And a dashboard, a Three.js digital twin of the greenhouse, so a room full of people can watch an actuator move on its own the moment the reconciler corrects drift, instead of reading it off a log line.
you ── commit greenhouse.yaml ──▶ hdm-sdcc-state (git, public)
│
│ clone once, fetch every cycle.
│ a failed fetch just keeps using
│ the copy already on disk
▼
reconciler (its own namespace,
poll loop one replica, no Service)
│
GET /state │ POST /actuators
(what is true) │ (only the drifted keys)
┌───────────────────┴───────────────────┐
▼ ▼
greenhouse device ◀── POST /actuators ── you / the audience
(HTTP API only) (dashboard buttons or a live curl,
│ the same write the reconciler makes)
│ GET /api/greenhouse/state (proxied, every 7s)
▼
3D dashboard (a digital twin, so drift is visible on screen)
´´´
All four repositories run on a three-node k3s cluster on one dedicated Hetzner server, behind Proxmox and reachable only over Tailscale, with inbound web traffic arriving through a single Cloudflare tunnel that makes outbound connections only, so the cluster has no open inbound port at all. Each piece is its own container image, built and rolled out by its own CI pipeline, and the deploy step runs on a self-hosted runner living inside the cluster as an ephemeral pod, authenticating with its own pod identity fetched at job time. No cluster credential ever ends up sitting in a GitHub secret.
The architecture I’ve planned originally
Here’s the part of this project I think is actually the most honest thing in it. The first real plan wasn’t the four-piece system above. It modeled the greenhouse as a Kubernetes Custom Resource Definition, Greenhouse, with a proper apiVersion, kind, spec, and status, registered with the cluster, watched by a hand-rolled controller with its own REST client, two separate Deployments mirroring how Flux splits a source-controller from a kustomize-controller, and RBAC scoped down to greenhouses and greenhouses/status as their own resources. It was consciously modeled on how Argo CD and Flux actually work under the hood, level-triggered reconciliation, single-replica-as-leader-election, informers standing in for what became a polling loop. None of that exists in the shipped code. The whole design got reversed before a single line of it was written.
Part of the reason is architectural, and I think it holds up on its own. A CRD plan puts two extra steps, registering the resource type and standing up a controller that maintains a status subresource, between “start building” and “anything reconciles.” Under time pressure that git half is exactly the thing that gets cut, and what’s left is a Kubernetes controller polling an HTTP API, which isn’t really GitOps anymore, it’s just a controller. Cloning git directly also makes the offline story real instead of just described. A git clone already is a full local copy of the desired state, so the reconciler keeps working from it when the network drops, no extra design needed. With desired state sitting behind a controller in etcd, that story gets fuzzier, since etcd is itself a remote dependency the moment you’re not running on the exact same node it lives on.
The more honest reason is that partway through, I noticed the CRD direction had quietly drifted toward a much bigger scope than the course actually needed, a real Kubernetes operator with its own API surface, without me really noticing it happening while it happened, because that’s the kind of thing that feels like the “proper” way to build this. Once I actually stepped back and compared what the project needed to demonstrate against what I’d started sketching, the gap was hard to miss. Reversing an architecture decision before anything is built is basically free. Doing it three weeks in, after wiring up a controller and a status subresource, would not have been. So the CRD isn’t a first draft I’m quietly hiding, it’s the thing this whole post is arguing against having built, which I think is a more useful thing to show than a clean success story would be.
It’s also not wasted work. Weighing it honestly against what got built instead is most of what the rest of this post does. And getting it precise mattered more than I expected, the rejected “compare a stored hash to a computed one” design isn’t actually well described by Kubernetes’ last-applied-configuration annotation, which is a stored full config used for a three-way merge, not a hash. Argo CD’s own “OutOfSync” detection, a live field-by-field diff against the git-declared manifest, is closer in spirit to the plain diff this project actually ships than to the CRD design it replaced. Small distinction, but it’s the kind of thing that’s worth getting right rather than gesturing at.
Comparing Flux, Argo, and the rest
Once the CRD was off the table, the honest question became what a from-scratch reconciler is actually trading away against the tools that already exist for this.
Flux and Argo CD are both the reference implementations of GitOps, and both work by reconciling Kubernetes objects against a git repository, cluster manifests, Helm releases, Kustomize overlays. Using either one here would have meant building the CRD anyway, since neither tool knows how to reconcile an arbitrary HTTP device API, only cluster resources. Argo CD adds a web UI, SSO, and an app-of-apps model for coordinating many applications across many clusters, none of which means anything for one greenhouse. Both are the right call the moment there’s a real cluster with many workloads and a team that needs RBAC and a shared sync dashboard, which this project deliberately isn’t.
A CronJob that starts a pod every few seconds, does one reconcile pass, and exits was the other serious alternative to the long-running Deployment that’s actually running. A fresh pod each run either re-clones the state repo every single time or needs a persistent volume to avoid that, either way it fights the durable-local-copy behavior that makes the offline story work at all. It’s also just a worse thing to demo live, the room watches one process notice and act, instead of a new pod appearing every interval doing the same thing once. Below the orchestrator layer entirely, there’s cron plus a plain git pull && ./apply.sh on a schedule, no Kubernetes in the picture at all, which is genuinely how a lot of small real GitOps setups work, it just loses restart-on-crash, resource limits, and any story about a fleet of these coordinating.
Then there’s the industry answer for actual edge fleets, device platforms like balena, Mender, Azure IoT Hub, or AWS IoT Greengrass, which bundle OTA updates, A/B partitions, fleet dashboards, and device identity into one managed product. Far too heavy for a single simulated device in a lecture demo, and they’d hide the reconciliation loop behind a product UI instead of showing it, which is the opposite of what this project is trying to demonstrate. Edge-extended Kubernetes, KubeEdge or OpenYurt, sits closer conceptually, letting an edge node keep running when its connection to the control plane drops, which is exactly the same idea as this reconciler’s cached local clone, just implemented one layer further down the stack.
The concrete version of “too heavy” is a resource footprint, and this is a number worth actually sourcing rather than guessing at, so I pulled it straight from each project’s own default values.yaml this month rather than repeating a round figure from memory. A quick note on what these numbers mean before the comparison. Every pod on a cluster can ask for a slice of a CPU core and an amount of memory before it’s even allowed to start, so these are directly comparable across projects even though they’re doing very different amounts of work. This reconciler asks for about 1/40 of a single CPU core and 64 megabytes of memory, next to nothing. Flux ships six separate controllers (source, kustomize, helm, notification, image-reflector, image-automation), each defaulting to a tenth of a core and 64 megabytes on its own, so the two you’d actually need for a plain git-plus-manifest setup, source-controller and kustomize-controller, already add up to a fifth of a core and 128 megabytes together, and the full chart with all six comes to three-fifths of a core and roughly 384 megabytes. Argo CD is the more interesting case. It’s Helm chart ships every component, the application controller, the server, Redis, Dex, with no CPU or memory floor set at all by default. That’s not a small number, it’s no number, the chart trusts the operator to size it themselves rather than publishing a default opinion the way Flux does. Worth naming as its own small data point about how differently two projects solving the same problem choose to ship.
Where GitOps stops working
This is worth more than the comparison above, honestly, because almost nobody shows the limits of their own design on purpose.
A reconciler protects against drift, reality diverging from what git says it should be. It does nothing at all about a bad declared state. If the file says the pump should be on and that’s actually wrong, a correctly working reconciler keeps the pump on anyway, forever, because matching the file is its entire job by definition. GitOps enforces intent, it never judges whether the intent was good.
The mechanism that actually catches a bad update comes from a completely different layer, and it comes from embedded systems rather than from GitOps at all. Just keep a spare, already-working copy of the software sitting on the device itself, and only switch over to a freshly updated copy once it’s actually proven itself. This is a real, established pattern in embedded and IoT device updates, not something specific to this project. The update lands on the device’s inactive slot, the device boots into it, and if a local self-test doesn’t pass within a bounded window the bootloader rolls back automatically, on its own, with zero dependency on git, the cloud, or the reconciler, because its entire job is to work exactly when those are the things that just broke.
Put the two layers together carelessly and they actively fight each other. The device rolls back to its last-good config locally. The network comes back. The reconciler sees drift again, since the device is running the old config and git still says the broken new one, and faithfully re-applies the exact update that just failed. That’s an infinite, self-inflicted crash loop, built entirely out of two systems each doing their job correctly. The fix is that a local rollback has to leave a mark, something like “this update failed, don’t retry,” that the reconciler is actually taught to respect before touching that field again. That mark is basically what Kubernetes’ status field is for at a conceptual level, not just the current state, but the reason it’s deliberately not matching spec right now.
A short honest list of everything else this design doesn’t cover. the reconciler’s local git clone lives on ephemeral disk, so a pod restart while the network happens to be down loses it, and there’s no desired state to act on until the connection returns, a gap that’s documented rather than papered over with a volume, since the real edge answer is a last-known-good baked into the image, not a single-node persistent volume that doesn’t even survive a node failure on this cluster. The device is only ever correct within one poll interval of any drift, never instantly, which is a designed trade-off rather than an oversight. There’s exactly one reconciler replica, so if it goes down nothing reconciles until Kubernetes notices and restarts it, fine at this scale, worth naming anyway. And the device trusts its network entirely, no auth on the write path at all, which is fine for a simulator behind a private cluster and would not be fine for a real one.
Is this a cloud computing project
Worth addressing directly, since the reconciler never calls the Kubernetes API and mostly just speaks plain HTTP to one device. I think the answer is yes, and the reasoning is that the Kubernetes API is itself an HTTP API, what makes it distinctive is its watch semantics and resource model, not the transport underneath. A client that reads desired state, reads observed state, diffs the two, and issues corrections over HTTP is doing structurally the same thing every Kubernetes controller does, just against a simpler surface. This project also runs a real multi-node cluster, ships three services as container images through CI/CD, uses workload identity for deployment instead of a stored credential, relies on cluster DNS for cross-namespace service discovery, and implements the reconciliation control loop, which is the defining cloud-native control pattern and the whole reason GitOps is a named practice out of that ecosystem in the first place, Flux and Argo are both CNCF projects. The fleet-scale and offline-tolerance discussion above is distributed systems at the edge, squarely the same field.
Takeaways
GitOps is a declared source of truth plus a loop that never stops closing the gap between it and reality. The loop is the part people forget when they first hear the term, without it what you’ve got is just version-controlled config, which is useful on its own but isn’t the same thing.
Level-triggered reconciliation is why that loop is actually reliable rather than just tidy. It only ever looks at the world as it is right now, so a missed cycle self-corrects on the very next one instead of needing to be caught up on.
At the edge specifically, the loop has to run locally and survive a link that isn’t always there. A git clone already is a durable last-known-good copy, for free, which is the real reason cloning git directly beat putting a controller and etcd in the middle of this project, not just that it was less code to write.
And GitOps enforces intent without ever checking whether that intent was any good. Catching a genuinely bad update needs a separate, local rollback layer with no dependency on the network at all, and getting that layer to cooperate with a reconciler that’s trying to enforce the opposite thing is a real, mostly-unsolved design problem, not something either side handles for free.
The reconciler correcting real drift on its own, live, the first time it was actually deployed, before I’d written a single word of this, is still the part I keep coming back to. Same idea as a light rig coming back up on its own mid-show, just with enough distance from the fix this time that a person walking over and flipping the switch back was never going to be the plan.
Repositories
You can find the repositories for each component of the project under:
1. State
2. Greenhouse
3. Reconciler
4. Dashboard
Deployed Live Demo
My project is live and usable. Feel free to test if you’re stronger then my reconciler by making my greenhouse drift over the dashboard
Leave a Reply
You must be logged in to post a comment.