Picture a Kubernetes cluster without a single network card. Its nodes know nothing of TCP/IP, or even of Ethernet; they call to each other over the radio in short 32-byte packets, like foremen on walkie-talkies at a building site. The control plane is written in a language from the late eighties and, network protocol included, takes about 1,250 lines. Starting a pod involves no download: the node simply loads a module and calls a procedure in it. And best of all, the whole thing opens in a browser tab, where you can pull a node’s plug, jam the air, slip the cluster a forged command and watch what happens.

I built this cluster over the last few weeks and called it Kube. It is a Kubernetes control plane written in Oberon, Niklaus Wirth’s language, and it runs on Oberon machines, that is, on the processor and operating system Wirth designed himself, from the circuit to the windows on the screen. Kube’s nodes are Oberon machines too, and they talk over a radio network from the same book: Wirth wrote that network for his workstations back in the late eighties, and the new edition of the project moved it onto a cheap radio module.

Let me say straight away why I am doing this, since in stories like this one that question always comes first, and usually with a smirk. For me this is not a project for laughs, nor an attempt to drag a pretty old technology into the light for people to admire and move on. It is research first and foremost. I want to understand what the infrastructure of the past might have become if the ideas we now call Kubernetes had appeared in the years when a computer was small, wholly understandable by one person, and connected to its neighbours by whatever was at hand. What interesting principles the forgotten technologies carried, and what we threw out along with them. What those technologies lacked, and why they were abandoned. And if you honestly sort out both, you can try to look into tomorrow and imagine an infrastructure where complexity need not grow faster than usefulness.

Kubernetes is a convenient point of reference here. Its main idea fits in one sentence: you do not tell the system what to do, you describe what should exist, and a set of independent loops compares the wanted with the actual over and over and closes the gap step by step. The idea knows nothing about Go, containers, etcd or clouds, and it can be carried over to any machine. Wirth’s machine suits it better than any, because it can be read in full, from the processor’s registers to the pod scheduler, with nothing hidden under a framework. When you move a big system into such a small one, it immediately becomes clear what in it is essence and what is sediment, and which decisions were forced and which were mere habit.

This is the second part of my series on paleocomputing. In the first part I told how I ran Wirth’s processor in a browser, in QEMU, in Kubernetes and in Cozystack, where an Oberon machine can now be installed from the catalog with one click. You do not need to read it: whatever concerns Oberon and Wirth’s machine I will explain along the way, and Kubernetes you probably know as well as I do. The article came out long again. First I will tell how Kube is built and how it works, then about the principles the Oberon language and operating system imposed on it, how it differs from real Kubernetes, where it turned out better and where noticeably worse. After that come six labs you can do right in the browser, instructions for those who want to install it all themselves, and at the end my conclusions about what such experiments can teach the people who build infrastructure today.

If you would rather touch first and read later, open the cluster lab. Nothing needs installing. Within half a minute three machines boot, the control plane starts Kube, the nodes join, and a cluster table appears on the right, which the page builds simply by listening to the air. Sources, documentation and every measurement are in the repository github.com/tym83/paleocomputing.

How Kube is built

Oberon and Wirth’s machine in brief

Oberon is both a programming language and an operating system. They were made in the mid-eighties at ETH Zurich by Niklaus Wirth, the author of Pascal and Modula-2, and Jürg Gutknecht. The language is tiny: modules, records, arrays, pointers, garbage collection and strict typing, while exceptions, generics and even unsigned integers are absent. The system matches the language. It has no processes, no threads and no memory protection. At the very bottom spins a single loop, Oberon.Loop: it polls the mouse and keyboard, calls commands one at a time, and in the pauses between input events calls background tasks, Oberon.Task. A task cannot be interrupted, so it must do its piece of work quickly and return. Multitasking here, in other words, is cooperative and rests on the good manners of everyone involved.

In 2013 Wirth published a new edition of the book Project Oberon and added his own processor to the system, RISC5, described in Verilog. Loaded into an FPGA, it ran on a board with a megabyte of memory at 25 MHz. That is Wirth’s machine. With us it lives in three guises: in the browser runs Wirth’s actual circuit, translated to C++ and compiled to WebAssembly; in QEMU runs our own model of the machine; and in Kubernetes the same QEMU model runs inside KubeVirt.

Where it all started

The first version of Kube was a pure toy, and I said so honestly in a separate episode of the series. It was a single Oberon module holding an object store and three controllers, for deployments, ReplicaSets and nodes. The nodes were three names, node-a, node-b and node-c, and a pod counted as running only because a controller had written a node’s name into it. Nothing ran anywhere. But even that version showed very clearly that the heart of Kubernetes is not containers but reconcile loops, and that these loops fit Oberon beautifully.

They fit because Oberon already has everything a control plane needs, namely a central loop that calls background work. Kube.Start installs the three controllers as three Oberon.Tasks with a period of 50 milliseconds, in the same ring where the system’s garbage collector spins, and from then on Oberon.Loop calls them whenever the user is not typing or moving the mouse. The controllers do not call each other or send each other anything. Each looks only at objects of its own kind and changes only what it owns, and the result of its pass becomes the input of a neighbour on its next pass. That is exactly how the controllers of real Kubernetes interact: through shared state, not through messages.

To call all this a cluster, two things were missing. First, nodes, meaning several machines and a way for them to talk. Second, rollouts, where one version of an application replaces another without downtime. With nodes I thought at first that I had hit a dead end, since Wirth’s machine has no network. Then I reread the book and found the radio in it.

Wirth’s radio

The Oberon workstations at ETH were always networked. Wirth wrote that network back in the late eighties for the wired Ceres workstation, and in the 2013 edition Paul Reed, who worked with him on the new version of the project, moved it onto radio. The board carries an nRF24L01+ transceiver, a cheap chip still found today in wireless keyboards and hobby gadgets, and the processor talks to it over SPI, a simple serial bus. The chip is very modest. It sends a frame of at most 32 bytes at a time, received frames wait their turn in a three-slot buffer, all stations on one channel hear each other, and there is no collision avoidance at all: if two stations speak at once, the receiver gets either garbage or nothing.

The chip’s driver, the module SCC, is about two hundred lines. Each of its packets starts with an eight-byte header holding addresses, a type and a length, and a long packet is cut into several 32-byte frames. On top of SCC sits the module Net, a small protocol stations use to send each other messages and files. From here on, by Wirth’s radio I will mean exactly this pair, and by the air everything the stations on one channel hear.

Why radio rather than an ordinary network? Because Wirth’s machine simply has no other. It has neither Ethernet nor a TCP/IP stack, and writing them would mean building a little Linux on Oberon. The radio, on the other hand, is described in the very same book, and I was curious what kind of system would come out if I honestly accepted the machine’s limits instead of dragging a modern network onto it.

A virtual machine has no real radio, of course. So our QEMU model emulates the whole nRF24L01+ chip, registers and queues included, and wraps every frame it sends in a UDP datagram. The datagrams flow to a relay, a small program that hands every frame to all the other machines. That is the air. The relay can lose a given share of frames, and if you stop it, the air is gone altogether. In Cozystack, our open platform built on Kubernetes, the relay became a catalog application, OberonAir, and in the browser the page itself plays the air.

Three messages, each in one frame

The protocol, which I called KubeNet, has just three messages. Each goes to everyone at once, since a radio cannot pick a receiver anyway, and each fits into one frame.

A node sends its heartbeat once a second, giving its name and the ids of the pods that are actually running on it right now. The control plane sends an assignment to every live node once a second, listing the pods bound to that node. The third message, a spec, also comes from the control plane and tells which image the pod with a given id has. Specs go out one per tick, pods not yet started first.

Why one frame and not a message of any length? SCC can send packets of up to half a kilobyte by cutting them into frames, but the air has no collision avoidance. If two stations start sending long packets at the same time, their frames get interleaved at every receiver, and a receiver can no longer tell whose frame is whose. One could invent air arbitration, queues and retransmissions, but that is exactly the road along which networks arrived at the complexity I wanted to get away from. So every message fits into one frame, which leaves 24 bytes of data after the SCC header.

In a heartbeat and an assignment these bytes are laid out as follows: the cluster tag, six bytes of node name, the pod count, up to ten pod ids at a byte each, two bytes of counter and four bytes of signature. Exactly twenty-four. Hence all of Kube’s odd limits: a node name is at most six characters, a pod id takes one byte, and a node runs at most ten pods, because no more fit into a heartbeat and the scheduler will not bind more.

The main property of the protocol is that it is level-triggered, like Kubernetes controllers, only not inside the control plane but on the network itself. A node runs exactly what its last assignment says, not a sequence of “start” and “stop” commands. If an assignment is lost, another with the same content comes a second later, so there is nothing to repair. If a heartbeat is lost, the control plane waits for the next one. There are no acknowledgements, no retransmissions and no sequence numbers for reliability: all of the reliability comes from each message carrying the full state rather than a change. Even on an air that loses 30 percent of its packets, not a single pod moved in a minute, and I checked that.

A pod is a module

In real Kubernetes a pod’s image is a file-system archive with a program inside: the kubelet pulls it from a registry and runs it in an isolated container. Oberon has nothing like that, but it has a mechanism that works similarly and much faster. A pod’s image in Kube is simply the name of an Oberon module on the node.

On learning of a new pod, the kubelet calls Modules.Load with the image name. If the module is not loaded yet, the system finds its compiled file on disk, loads it into memory, links it with every module it imports, checks the keys of their interfaces and runs the module’s body. A key is a kind of checksum of the interface that the compiler writes into every module. If an interface has changed, the system refuses to load modules compiled against the old version. Then the kubelet finds the module’s Start command and calls it, and when the pod is to go away, it calls the same module’s Stop command.

A command in Oberon is any exported procedure without parameters. You run it by middle-clicking the text Module.Procedure in any window, and if it needs parameters, it reads them itself from the text after its name. So a pod’s id cannot be passed directly, and that is what a tiny module, Pods, with two variables, the id and the image, is for. The kubelet fills them in before the call, and the workload module reads them. Not the most elegant solution, but very much in Oberon’s spirit: a module’s global variable is a legitimate way to pass context here, because only one command runs at any moment and races have nowhere to come from.

The teaching workload comes in two versions, Ticker and Ticker2. For each pod it starts it keeps a seconds counter, and the command Ticker.Show prints which pods run on this machine and how many seconds each has lived. They make rollouts easy to watch: the node’s log shows the first version’s pods stopping and the second version’s starting.

If there is no module of that name on the node, Start cannot be called, and the kubelet simply leaves the pod out of its heartbeat. The control plane sees that the pod is assigned but not running, and it stays Pending. In Kubernetes, that is what a pod whose image could not be pulled looks like.

The store on disk

In Kubernetes the whole state of the cluster lives in etcd, and a restarted control plane reads it from there. In Kube the role of etcd is played by an array of 256 records in the control plane machine’s memory, and so that it survives a restart, a background task writes it to disk whenever it changes.

It is written to two files in turn, Kube.Store0 and Kube.Store1, each holding a generation number and a checksum. If power fails in the middle of a write, only one file is damaged, and the other, of the previous generation, survives. At start Kube.Start reads both and takes the newer of the whole ones. The files are rewritten in place rather than created anew, and Oberon demands this too: its file system frees the space of replaced files only at the next boot, and if a new file were created for every change, a busy cluster would simply run the disk out.

After a restart the control plane gives the nodes one heartbeat timeout to report, and only then begins to count them NotReady. Kubernetes does the same after its node controller restarts. The start of that timeout, by the way, had to move from the moment the store has been read to the moment the control plane starts listening to the air. In the cloud, a person managed to type something over VNC between those two commands, the timeout ran out, and every pod moved, although none had stopped.

Rollouts, and two bugs Kubernetes knows well

The command Kube.Apply web 6 Ticker2, given to a running web 6 Ticker, changes the image. The deployment controller creates a new ReplicaSet and starts moving pods one at a time. First a pod with the new image is added. As soon as its kubelet reports it running, there is one pod more than wanted, and the old ReplicaSet removes one of its own. This repeats until the old ReplicaSet is empty, and then it is deleted.

It looks simple, but two bugs surfaced along the way, and both turned out to be old acquaintances of Kubernetes.

The first concerns stopping a pod. When the control plane deletes a pod from the store, the pod keeps running on its node until the kubelet hears the next assignment, which can take up to a second. By then the controller already sees one pod fewer and adds a new one. As a result, for a moment eight pods ran where seven were allowed. Real Kubernetes behaves in exactly the same way, and only recently has the Deployment gained the field podReplacementPolicy, which makes it wait until old pods have stopped completely, and even that is still alpha, behind a feature gate. In Kube I made this the only behaviour: pods the nodes still report but the store no longer holds count as stopping, and while there are any, no new pod is added.

The second bug concerns pod ids. A node knows a pod only by its one-byte id. At first a new pod got the lowest free id, and that often turned out to be the id of the very old pod it had just replaced. The kubelet saw a familiar id in the assignment and decided nothing had changed. The store claimed Ticker2 was running, while Ticker kept spinning on the node. This is exactly why Kubernetes never reuses a pod’s UID. Kube’s ids now go round, and the check after every rollout requires that no id of an old pod is running any more.

The failures clusters are built for

A cluster exists to survive failures, and I decided to test that the way I would test a real one. A separate test starts a control plane and two nodes and judges them solely by the air. For that a passive listener joins the relay: it sends nothing, only records every heartbeat and every assignment and checks their signatures. The test asks Kube itself almost nothing: only the outcome of a rollout is compared with what Kube wrote to disk, and everything else is judged by what actually happened on the air.

Switch a node off, and after five seconds of silence the control plane marks it NotReady, and two seconds later moves its pods to another node, so that the whole thing takes about eight seconds. Switch the control plane off, and the nodes keep running their last assignment, because there is nobody to tell them otherwise. When it boots again, the store comes back from disk and not one assignment changes. If the nodes lack the module, the pods stay Pending. And if 30 percent of packets are lost for a whole minute, not one pod moves.

At first the timeout was three seconds. On an air losing 30 percent, three heartbeats in a row went missing several times a minute, and the control plane, taking the node for dead, moved pods that had never gone anywhere. With five seconds, false alarms became dozens of times rarer, and the price was slower recovery from a real failure. Kubernetes makes the same bargain at another scale: the kubelet reports every ten seconds, and the control plane waits forty to fifty.

The most instructive case was losing the air. When the relay falls silent for twenty seconds, all the nodes fall silent at once. A control plane that simply trusts its timeouts decides the nodes have died and tries to move every pod, although there is nowhere to move them. And when the air comes back, the nodes come to life one by one: the first one back receives everyone else’s pods, the second takes some of them back, and so on. In the first version twenty seconds of outage changed 74 assignments. The nodes kept working the whole time and noticed nothing, while the cluster staged a disaster for itself.

Kubernetes knows this trap: when all nodes of a zone go NotReady at once, the node controller puts the zone into the FullDisruption state and stops evicting pods, reasoning that all nodes dying at the same moment is far less likely than a lost link. Kube does the same when all nodes, or at least 55 percent of three or more, go NotReady. But it did not work at once: it took two details that only came to light in a failed check.

First, nodes do not go NotReady at the same moment; their heartbeats are up to a second apart. So the first node to go silent lost its pods before the others went silent and it became clear that this was an outage. Now a node’s pods move only after it has been NotReady for two seconds, and by then all the others have gone silent too. Kubernetes waits five minutes for this by default. Second, nodes also come back one by one, and the very first to return took the cluster out of that state before the others had reported, so their pods moved at once. Now the silent nodes get one more timeout to report after leaving it, as Kubernetes does too, resetting the nodes’ timers when a zone leaves FullDisruption. With these two corrections, twenty seconds of outage change not a single assignment.

Strangers on the air

Once, in the sandbox, two clusters ended up on one air, and both had a node of the same name. A node of one cluster kept starting and stopping its pods, because it obeyed the assignments of both control planes in turn. So a cluster tag was added to the start of every message, a byte computed from the cluster’s name, and a node began to ignore messages with a foreign tag.

But a tag proves nothing; anyone can send it, so a signature came next. Every message carries an authentication code, a MAC, computed from the message itself and the cluster’s secret key, and without the key the right code cannot be guessed. It is computed with HalfSipHash-2-4, a reduced variant of SipHash that works on 32-bit words and gives a 32-bit result. HMAC-SHA256 will not do here: Wirth’s processor is 32-bit and runs at 25 MHz, and of the message’s 24 bytes the signature can take four at most. HalfSipHash was made precisely for such small devices, and in Oberon it takes about thirty lines. Thirty-two bits is on the small side for serious cryptography, but for a teaching cluster it is an honest compromise.

A signature does not prevent a recorded genuine message from being replayed, so every message also carries a 16-bit counter, and the receiver drops anything not newer than the last one accepted from the same sender. The failure test checks this too. Right after a genuine assignment it sends the node, on behalf of a stranger, “run nothing” in three ways: tagged as another cluster’s, signed with the wrong key, and as a genuine assignment recorded earlier. The node ignores all three. The same message, honestly signed with the key and with a fresh counter, it obeys, and that is the control case without which the check would prove nothing.

How much it takes

A separate load test starts a control plane and two to eight nodes, three pods per node, and measures from the air how fast the cluster converges and how fast it recovers from losing a node. At any size it converges in about three seconds, and losing a node costs about eight: five seconds of timeout, two seconds of pause before eviction, and up to a second until the next assignment. There was never a false NotReady.

The cluster does have a ceiling, and it lies not in the radio itself but in Wirth’s driver. Before every send, SCC waits 50 milliseconds to let someone else’s acknowledgement finish, and the machine does nothing else meanwhile. So one station can send at most twenty messages a second. The control plane needs one assignment per node every second, so with eight nodes it spends almost half its time waiting, while received frames pile up in the three-slot buffer and the extra ones are lost. During a rollout specs join the assignments, one per tick, and the waiting takes up almost all the time. By my reckoning the protocol can carry about twenty nodes when idle and about ten during a rollout. That is a calculation from the code, not a measurement. Besides, every Oberon machine occupies a whole host core, since its loop never idles, and eight nodes on one machine load that machine rather than the air.

Two bugs in our QEMU

To make all of this work, two bugs had to be fixed not in Kube but in our machine model for QEMU, and without the cluster I would have found neither. The MOD operation after a multiplication sometimes returned the wrong half of the product, so the store’s checksum always came out as zero. The background task saw no changes in it, and the store was never written once. And a machine that had once lost its relay stopped hearing the air for good, because QEMU’s UDP channel quietly dropped its reader after a failed read. Both findings are described in the repository, and they are a good example of why I so like running real programs on an emulator: booting the system triggered neither bug.

Commands at start, and OberonKube

The last step was about convenience, but without it everything else would have lost its point. Nobody will start a cluster a second time if commands have to be typed over VNC on every machine. I needed each machine to know who it is at start, the way a cloud VM learns its role from cloud-init.

QEMU now takes a string of commands for the machine to run at start and feeds it to the serial port, as if it had been typed at a console. A small module, Boot, reads the string and runs the commands one after another, each with its parameters. It is called at the very end of the body of System, the module loaded when the system starts. System itself is rebuilt from the image’s own sources with that single line added, so its interface, and the key every other module checks, stayed the same.

Here the cluster taught one more lesson. The commands at start run at every boot, not just the first. If Kube.Apply web 4 Ticker is among them, then after a restart of the control plane the deployment returns to what the commands say, and a rollout done by hand since then is quietly undone. Whether that is good or bad depends on what you take as the source of truth. In the cloud it is the order form, and returning to it is exactly the right behaviour there. But in the browser lab, where a person rolls out a new version by hand, it was a bug, and it was the lab itself, by the way, not the QEMU tests, that found it. For such cases there is now the command Kube.Ensure, which creates a deployment only if there is none yet.

In Cozystack all of this came together in one catalog application, OberonKube. A user says how many nodes they need, the cluster key and a list of deployments, and the catalog creates the air, the control plane machine and the nodes, and gives each machine the commands for its role. The cluster forms by itself. The form here is the source of truth: a changed list of deployments takes effect at the next restart of the control plane. The machines of one air also try to land on different hosts, so that losing a host costs as few nodes as possible.

What Oberon imposed

When you write Kubernetes in Go for Linux, almost any decision can go any way. Need a queue, and channels are at hand; need a database, and there is etcd; need a network, and there is gRPC over TCP; need isolation, and there are kernel namespaces. There is always a choice, and so decisions are often made out of habit. On Oberon there is almost nothing to choose from, and that is the most interesting part: every limit of the language and the system forced a quite specific decision, and those decisions show clearly which properties of Kubernetes follow from its very idea and which from what it was built on.

I will split the limits into those that came from the language and those that came from the operating system and the machine, although with Wirth the boundary between them is a convention: it was all done by one hand.

What the language imposed

Sizes are known in advance. In Oberon-07 an array’s size is nearly always known at compile time, and dynamic memory is allocated only for records reached through pointers. Writing a program in such a language where everything grows as needed is awkward, and I did not try. Kube’s store is an array of 256 objects, a node has at most ten pods, a node name is at most six characters, an image name at most fifteen, a pod id takes one byte. Each of these numbers has a reason, and all of them sit in two blocks of constants at the top of the modules Kube and KubeNet. Thanks to this, Kube allocates no memory in its working loop, cannot exhaust it under a flood of objects, and behaves predictably at its limits: an extra pod simply will not be created. Real Kubernetes lives with limits too, such as 110 pods per node by default or a megabyte and a half per object in etcd, but there they are scattered across the documentation, while here they cannot be hidden.

Integers are 32-bit, and only bytes are unsigned. HalfSipHash works on exactly 32-bit words and produces a 32-bit signature that fits in the frame. Hence also the counter arithmetic modulo 65,536, with a careful “is this number newer” check that stays correct after wraparound. Hence too the store’s checksum, computed so as not to depend on sign. A small thing, but it is exactly where the bug with MOD in our QEMU turned up.

Commands without parameters. An exported procedure without parameters counts as a command, and one with parameters does not. So the kubelet cannot call Start(pod): it puts the pod’s id into the Pods module and calls Start with no arguments. In any other language this would be considered bad form, but here there is no other way, and it is safe, because only one command runs at a time.

Modules with keys. Every compiled module carries the key of its interface, and at load time the system checks it against what the importing modules were compiled with. So a pod’s image in Kube is not just a name but a name with a built-in compatibility check. If a workload module was compiled against an old version of Pods, the system refuses to load it, and the pod stays Pending instead of crashing in the middle of its work. In the container world the closest thing is a pinned image digest, but that only guarantees that you got exactly those bytes, not at all that they are compatible with their environment.

The language is small. It sounds like a drawback, but in practice it disciplines more than anything else. Oberon has no generic collections, exceptions, interfaces, goroutines or reflection, so all of Kube is written as plain loops over arrays and procedures that return a boolean instead of throwing. It reads almost like pseudocode, and that is just how I wanted it to read.

What the system and the machine imposed

One loop and cooperative tasks. This is the main one. Oberon has no threads, so Kube’s controllers run as tasks in the central loop, and the kubelet on a node is a task too. Each task does a short piece of work and returns, and three things follow at once. First, Kube has not a single lock, mutex or channel, because races have nowhere to come from: while one controller runs, the rest stand still. Second, behaviour is deterministic: given the same input, the controllers do the same things in the same order, and such a system is far easier to debug. Third, and this is a minus, any task that stops to think for long freezes the whole machine, kubelet and radio included. A pod whose Start command goes into an endless loop hangs the entire node.

No memory protection. All modules live in one address space, and the only thing that keeps one module from corrupting another’s memory is a strictly typed language with array bounds checks. For Kube this means that pods are in no way isolated, either from each other or from the kubelet. This is the most serious difference from real Kubernetes, and I will come back to it when we get to what Oberon lacked.

The interface is text. In Oberon any line of the form Module.Command in any window is a command. So Kube has no API server, no YAML and no kubectl. Its API consists of commands such as Kube.Apply web 6 Ticker2, Kube.Get and Kube.DeletePod, which a person writes in any window and runs with a middle click, reading the result in the system log. The same principle made the commands at start possible: a machine’s role is set by an ordinary line of commands that QEMU feeds to the serial port, and the module Boot runs it exactly as a person would. No special configuration format was needed; the configuration of a machine became the text of its commands.

An old-school file system. Oberon frees the space of replaced files only at the next boot. So Kube’s store is rewritten in place, in two files in turn, rather than created anew at every change, and that also protected it from a power failure in the middle of a write. A solution databases arrived at decades ago appeared here by itself, out of a file-system limitation.

Radio instead of a network. I have already covered this in detail: a 32-byte frame, broadcast to everyone, no collision avoidance. Hence a protocol of one-frame messages, full state instead of acknowledgements, a cluster tag and a signature in every message. And hence the most unexpected property of the whole system, which the next section is about: everything the cluster knows can be heard on the air.

Twenty-five megahertz. The machine’s speed set the periods: controllers fire every 50 milliseconds, the kubelet every 100, a heartbeat goes out once a second. The control plane needs very little actual computation, and nearly all its time goes into waiting. But alas, the waiting is not free either: before every send the radio driver spins in a loop for fifty milliseconds, and it is this waiting, not computation, that limits the size of the cluster.

How Kube differs from real Kubernetes

To create no illusions, here is a brief comparison. Kube embodies the idea of Kubernetes, not Kubernetes itself, and the difference between them is enormous.

This table shows that Kube is good for nothing Kubernetes is good for. But it shows something else too: all the logic Kubernetes exists for, that is, the wanted state, independent reconcile loops, rollouts without downtime, recovery from a lost node and caution when the link is lost, fit into a small system with almost nothing listed in the middle column. Everything else in Kubernetes is there so that this logic works on thousands of machines, for thousands of users and for applications nobody trusts. Very important things, but not the foundation.

Where Kube turned out better, and where Oberon fell down

Comparing a toy cluster on the radio with the Kubernetes that half the internet runs on seems unfair, and I am not going to claim Kube is better. What interests me is something else: which qualities emerged in Kube by themselves, with no effort at all, simply because it grew up on Oberon. I mean the properties that ordinary Kubernetes either lacks or pays far too much for. I counted six of them.

The whole cluster can be read in an evening

The control plane is about eight hundred lines, the network module with the kubelet and signatures about four hundred, and with the teaching workload and the commands-at-start module it comes to about fourteen hundred. Below lies only the Oberon system, which can also be read in full, and Wirth’s processor in Verilog, for which a couple of evenings will do. So the whole path from “I want six replicas of Ticker2” to the register of the radio chip through which an assignment flies off to a node can be followed by eye, without once hitting a library that nobody has ever read.

With Kubernetes that stopped being possible long ago, and not because it is badly written: it solves a huge number of problems Kube simply does not have. But in the end even an experienced engineer who has run it for years usually knows how it behaves but not why, and finds out only during an outage. In Kube the answer to any “why” is in one procedure.

The air is the observability

I did not plan this property; it arose by itself, and I like it best of all. Since every message goes to everyone and carries the full state rather than a change, at any moment the air carries everything the cluster knows about itself: which nodes are alive, what runs on each, what is bound to each, and which image each pod has. A passive listener that sends nothing reconstructs the whole picture of the cluster without asking it a single question.

All my checks work this way. Neither the failure test, nor the load test, nor the browser labs ever ask the control plane what is going on. They listen to the air and compare what actually ran on the nodes with what was assigned to them. A bug in Kube itself cannot fool such a check, since it looks not at Kube’s reports but at the behaviour of the nodes. In ordinary Kubernetes the same takes metrics, logs, audit, exporters and a separate system to collect them all, and still you see only what the components chose to tell about themselves.

A pod starts in the blink of an eye

In Kubernetes starting a pod means pulling an image, unpacking layers, creating namespaces and starting a process, and that takes seconds, or even minutes with a cold cache and a big image. In Kube starting a pod means loading an Oberon module, if it is not loaded yet, and calling its command, and if the module is already in memory, starting a pod comes down to a procedure call. The whole cluster converges in three seconds, and almost all of that time goes into waiting for the next heartbeat, not into work.

This speed was paid for with isolation, of course, so the comparison is unfair. But it shows where the time really goes. When people talk today about cold starts, about WebAssembly on the server or about V8 isolates, this is exactly the question: can a unit of deployment be made so small and so module-like that starting it costs as much as a function call, without giving up isolation?

Compatibility is checked at load time

I wrote about this in the section on principles, but it bears repeating, because it is a strong idea. A pod’s image in Kube is a module with an interface key, and the system refuses to load it if it was compiled against an incompatible version of what it imports. A pod with an incompatible image will not crash after an hour of work on an unexpected call: it simply will not start, it stays Pending, and that is visible at once. In the container world nobody checks an image’s compatibility with its environment except your tests, and the compatibility of images that call each other is checked by an API schema at best, and only if you have one.

No locks, and therefore no race conditions

Kube’s code has no mutexes, no channels and no atomic operations. Controllers, the kubelet, writing the store to disk and receiving radio packets all run as tasks of one loop and execute strictly in turn. So data races, deadlocks and bugs that show up once in a thousand runs have nowhere to come from in Kube. Behaviour is deterministic: run the same scenario twice, and the controllers do the same things in the same order.

There is a flip side, covered a little further on, but the very thought that a control plane need not be multithreaded seems underrated to me. The control plane of eight nodes needs very little computation. The control plane of a thousand nodes needs more, of course, but there too most of the time goes not into computing but into waiting for the network and the disk, and single-threaded event loops proved long ago that such waiting can be served without threads.

Security from the first message

Early Kubernetes left a lot open by default; the kubelet, for one, accepted anonymous requests for a long time, and closing such holes took years. In Kube the signature and replay protection appeared before the cluster learned to do anything useful, simply because the radio left no choice: on the air anyone can say anything, and that is obvious from day one. The signature is 32-bit, which is too little for the real world, but architecturally the protocol is right: every message is authentic and belongs to its cluster, and the check costs about thirty lines.

The whole cluster in a browser tab

And the last one, not about architecture but about what follows from it. An Oberon machine is so small that three of them, together with the air, fit into a browser tab. So everything I describe can not only be read but repeated without installing anything: switch off a node, jam the air, slip in a forged assignment. Kubernetes has good learning sandboxes, but they are always someone’s cluster somewhere in a cloud, while here the cluster lives on your computer and can be broken any way you like without bothering anyone.

What Oberon lacked

Now about where Oberon proved too tight, and why such systems most likely lost. For the research this part matters no less.

Isolation. This is the main one. In Oberon all modules live in one address space and trust each other. A pod that corrupts memory through SYSTEM.PUT corrupts the kubelet, the radio and everything else along with it. A pod that goes into an endless loop freezes the whole node, because a cooperative task that does not return stops the entire system. As long as the machine runs code by one author who trusts that code, all is well, and Wirth designed his system just that way: for one person at one computer. But a cluster exists precisely to run other people’s code, and without isolation that is impossible. Kubernetes on Linux gets isolation from the kernel, while to get it Oberon would need memory protection in the processor, preemptive multitasking and resource accounting, that is, it would have to become a different system entirely.

Preemption. Besides isolation, the control plane cannot interrupt a pod that computes for too long, nor share processor time among pods. No quotas, no priorities, no limits, only good manners. A teaching workload that adds one to a counter once a second does not care, but any real workload will stumble on this at once.

A network. Radio with 32-byte frames is excellent for control, but there is no room in it for application data. Kube’s pods have no addresses, no services and no way to talk to each other. Files can be sent over Wirth’s radio, but that would be more like a USB stick than a network.

Flexible sizes. The limits I praised for predictability turn into a ceiling: ten pods per node, 256 objects in the store, six characters per name and twenty messages a second per station. Each of these ceilings can be raised, but not without end, because they all follow from the 24-byte frame, static arrays and a simple driver. Kubernetes paid for having no such ceilings with enormous complexity, and looking at Kube you understand that this price was deliberate.

A highly available control plane. Kube has one control plane machine. If it dies for good along with its disk, the cluster is left without a master. The nodes keep running their last assignment, which is a good property, but the control plane cannot be replaced by another machine, because there is no consistent store across several machines. Raft can be written in Oberon, but over a radio without delivery guarantees and with 32-byte frames that would be a big piece of research of its own.

Multiple users. Oberon has one user, Kube one key per cluster. No namespaces, no roles, no separation of rights, and whoever knows the key can do anything. Kubernetes spends most of its complexity precisely on letting many people and teams safely share one cluster, while Kube has no such layer at all.

Put all these points together, and it becomes clear that Oberon lost not because it was badly designed, but because it was designed for another world, where a computer belongs to one person, all the code on it is written by that person or by people they trust, and a network is a way to pass a file to a neighbour. As soon as computers became shared and code became foreign, isolation, preemption and rights were needed, and simple systems gave way to complex ones.

Six labs in your browser

At first I doubted the cluster could be run in a browser at all. The Oberon machine in the labs of the previous article already ran in a tab: it is Wirth’s actual circuit, translated to C++ and compiled to WebAssembly. But it had no radio, and a cluster needs three machines that hear each other. So the browser machine got the same nRF24L01+ transceiver model as the QEMU one, and learned to hand the frames it sends to the outside and to take in other machines’ frames. Each machine runs in a background thread of its own, and the page takes their frames and hands them to the others, that is, the page itself is the air. It can also lose a given share of frames, cut a single machine off the air and read Kube’s messages, checking their signatures, which is why a cluster table built from the air appears on its right.

A serial port was needed too, so that machines learn their roles at power-on, as in the cloud. The control plane receives the commands Kube.Start, KubeNet.Serve kube 00c0ffee00c0ffee and Kube.Ensure web 4 Ticker, and the nodes KubeNet.Join with their names, so nothing has to be typed. Kube.Ensure is there for a reason, but more on that in the sixth lab.

Speed turned out to be a task of its own. In WebAssembly Wirth’s circuit executes under a million instructions a second per machine, that is, dozens of times slower than the 25 MHz board. Kube, meanwhile, counts time in machine seconds: a heartbeat every second, a timeout of five. Had I kept the machines’ clocks honest, the cluster would take several minutes to form, and the lab with a node switched off would take about five minutes. So the page runs the machines’ clocks ten times faster than their instructions, and the machines believe they run at 2.5 MHz. Their seconds pass at a pace the eye can follow, and the protocol does not care: it has no idea how many instructions fit into a second.

Background tabs held another unexpected trap: the browser slows timers in them sharply, and the cluster nearly froze. So the machines now spin their own loop and do not depend on the page’s timers. The last trap was input. The page types a command into a machine with keys and mouse clicks, and at first it did so in one go. That is about eleven million instructions, which with the fast clock is over four seconds of machine time during which the machine does not hear the air. After every button press the control plane declared both nodes NotReady, and all that saved it was that eviction stops in FullDisruption. Now input goes in small portions interleaved with the machine’s work, and the air is delivered in between.

Each lab checks itself and shows a tick when what it describes has really happened on the air. You cannot press a button and get a tick: the check looks at heartbeats and assignments, not at what you did.

Open the lab

1. The cluster forms by itself

Nothing to do but wait. The machines boot, and the control plane’s screen shows the module Boot running the commands that came over the serial port. Kube installs three controllers, starts listening to the air and creates the deployment web with four pods, and a couple of seconds later the lines “node node1 Ready” and “node node2 Ready” appear in the log.

Meanwhile the nodes show the kubelet receiving its assignment and starting the pods of the Ticker module.

On the right of the page the cluster table fills in. For each node it shows when it was last heard, which pods it says it runs, and which are bound to it. If those two columns differ, the cluster has not converged yet, or something has gone wrong.

2. Roll out new code

Press the button web 4 Ticker2. The page types Kube.Apply web 4 Ticker2 ~ on the control plane and runs it with a middle click, as a person would. Kube creates a new ReplicaSet and starts moving pods one at a time. The check passes the rollout when every pod runs the new image, and only if during all that time the number of pods never fell below four or rose above five. It counts from heartbeats, that is, from what the nodes actually ran.

When the rollout is over, press Ticker2.Show on a node, and it shows the new version counting. The whole story is in the node’s log: the Ticker v1 pods stopped, the Ticker v2 pods started, and the new pods have ids different from the old ones.

3. A node dies

Press Power off above the screen of a node that runs pods. Its screen goes dark, its heartbeats stop, and after five machine seconds the control plane marks the node NotReady, and two seconds later moves its pods to the remaining one. Switch the node back on, and it boots, joins the air and becomes Ready, but nobody gives its pods back: like the real scheduler, Kube does not move running pods to a node that came later. New pods from scaling up, however, will go to it as the least loaded node.

4. The air dies

Press Switch the air off and wait half a minute. All nodes go NotReady at once, but Kube decides that the link broke, not the nodes, and moves nothing. Switch the air back on, and a couple of seconds later the same pods run on the same nodes. The lab passes only if not one assignment changed during the outage or after it.

The most interesting part here is to open “what is going on” under the lab and read about the 74 reassignments the first version made in twenty seconds of outage. And if you like, switch the air off for a couple of seconds, shorter than the timeout, and see that nothing at all happens.

5. An intruder

The section “An intruder on the air” has four buttons. Each sends, on behalf of a stranger, the assignment “run nothing” to the node that runs pods, and does so right after a genuine assignment from the control plane, so that the forgery is the last thing the node heard. The first button tags it as another cluster’s, the second signs it with the wrong key, the third replays a genuine assignment recorded earlier. The node ignores all three and keeps running its pods.

The fourth button signs the forgery with the real cluster key and a fresh counter. That is the control case, and the node obeys, stopping its pods. Without it the lab would prove nothing: what if the node simply listens to nobody but the control plane? A second later the control plane sends its next assignment, and the pods come back, because the protocol carries the full state, not commands.

The forgeries are visible in the air log too, by the way: the page checks signatures itself and marks messages with a foreign tag or a bad signature.

6. The control plane restarts

Switch the control plane off and, a few seconds later, back on. It boots, runs its commands once more, reads its store from disk, and not one pod moves. The nodes kept running their last assignment all along, and the control plane, once back, gave them one timeout to report, and they all did.

It was this lab that found the last bug before publication. The commands at start first had Kube.Apply web 4 Ticker, and after a restart the deployment went back to Ticker, undoing the rollout from the second lab. The QEMU tests did not notice, because they restarted the control plane before any rollout. Here, where a person rolls out a new version by hand, the right thing is to keep what they did, so the commands now hold Kube.Ensure, which creates a deployment only if there is none yet. In the cloud, on the contrary, the order form is the source of truth, and Kube.Apply stayed there.

What else to try

The page does more than the labs. With the loss slider you can spoil the air, say, lose 30 percent of frames, and see that pods do not move anywhere. Push the losses much higher, and sooner or later five heartbeats in a row will go missing and Kube will take a live node for a dead one; that is the price of a timeout. Cut off the air isolates a machine without switching it off, and you can watch the cut-off node keep running its pods although the control plane has already handed them to others. That is the very case of a pod running in two places, and Kubernetes lives with it in just the same way. In the command field you can type any Oberon command, for example Kube.Apply api 2 Ticker ~ to create a second deployment, or you can click right into a machine’s screen and work in it by hand. The middle button there is a click with Alt.

How to try it yourself

There are four ways, from very simple, where nothing needs installing, to your own cloud. Everything below is open: the sources are in the repository tym83/paleocomputing, Kube’s code in the impl/kube folder, and a detailed description with measurements in impl/kube/README.md.

In a browser

Open the cluster lab and wait half a minute. Nothing needs installing: everything runs in the tab, and the server only serves files. The page is happiest in a recent Chrome or Firefox on a computer with several cores, since each of the three machines takes a core of its own. Keep the tab in view: in the background the browser slows it down, and the cluster, though it does not stop, lives noticeably slower.

To run the same page locally, clone the repository, start any static server in the impl/web folder, for example python3 -m http.server 8765, and open http://127.0.0.1:8765/kube.html. And if you need no browser at all, the same three-machine cluster runs in Node.js with node impl/web/kube-test.mjs. It boots three machines on Wirth’s actual circuit, waits for the cluster to form, and checks from the air that all pods run and all signatures are valid. The machines’ clocks are honest here, so it takes about a minute.

In QEMU on your computer

You need Docker, git and make. First build our machine model for QEMU. The build runs in a container, so QEMU’s dependencies stay out of your system, but it takes about ten minutes:

git clone https://github.com/tym83/paleocomputing
cd paleocomputing
make -C qemu build

The system disk, with Kube, KubeNet, the teaching workload and the commands-at-start module already compiled onto it, is easiest to take from the published image. Each machine needs its own copy of the disk, grown to eight megabytes so the system has room to write its store:

docker create --name payload ghcr.io/tym83/paleocomputing/oberon-run:v0.1.21
docker cp payload:/opt/oberon/payload/prom.bin .
docker cp payload:/opt/oberon/payload/oberon.dsk .
docker rm payload
for m in plane node1 node2; do cp oberon.dsk $m.dsk; truncate -s 8M $m.dsk; done

Next you need the air, that is, a Docker network with a relay in it:

docker network create kube-air
docker run -d --name relay --network kube-air -v "$PWD/qemu/radio:/r" \
  qemu-build:risc5 'python3 -u /r/relay.py'

And finally three machines, each with its commands at start. The string after commands= is exactly what the machine runs after booting, the commands separated by semicolons:

run() {
  docker run -d --name $1 --network kube-air -p 127.0.0.1:$2:5900 \
    -v "$PWD/.qemu-work:/src:ro" -v "$PWD:/w" -w /w qemu-build:risc5 \
    "/src/build/qemu-system-risc5 -machine 'oberon,radio=air,commands=$3' \
     -bios prom.bin -drive if=none,id=sd0,file=$1.dsk,format=raw -vnc :0 \
     -chardev udp,id=air,host=relay,port=7524,localaddr=0.0.0.0,localport=7524"
}
KEY=00c0ffee00c0ffee
run plane 5900 "Kube.Start;KubeNet.Serve kube $KEY;Kube.Apply web 4 Ticker"
run node1 5901 "KubeNet.Join node1 kube $KEY"
run node2 5902 "KubeNet.Join node2 kube $KEY"

This uses Kube.Apply, not Kube.Ensure as in the browser lab: release v0.1.21 does not have Kube.Ensure yet. The difference shows only if you roll out a new version by hand and restart the control plane: with Kube.Apply the deployment goes back to what the commands say.

The machines’ screens are on VNC ports 5900, 5901 and 5902. Booting in software emulation takes up to a minute, after which the control plane’s log shows lines about the nodes being ready, and the nodes’ logs about the pods started. Oberon’s mouse has three buttons, and the middle one runs the command it points at. So Kube.Get in any window of the control plane shows all objects, and Ticker.Show on a node shows its pods. To roll out a new version, write Kube.Apply web 4 Ticker2 ~ on the control plane and middle-click that line.

You can listen to the air from the side, as the tests do. The listener joins the relay as one more machine, sends nothing, and with --json prints every heartbeat and every assignment together with whether its signature is valid:

docker run --rm -it --network kube-air -v "$PWD/qemu/radio:/r" \
  qemu-build:risc5 'python3 -u /r/listen.py relay --json --key 00c0ffee00c0ffee'

And then you can break things. docker stop node2 switches a node off, docker stop relay jams the air, and docker start brings either back. If the relay gets a different address after a restart, the machines will find it themselves: after a failed send our model looks the address up by name again. The same folder holds inject.py, which can play the intruder.

The checks in the repository start the cluster in just this way, only automatically. After building QEMU and the tools with make -C impl tools, you can run them yourself: python3 qemu/test/kube_dr_check.py checks every failure discussed above (their table is in impl/kube/README.md), python3 qemu/test/kube_boot_check.py checks that the cluster forms from commands at start alone, and python3 qemu/test/kube_load_check.py --nodes 4 measures load. These checks build the disk from source, so they already have Kube.Ensure. Keep in mind that each Oberon machine takes a whole core, since its loop never idles, and eight nodes on a laptop will measure the laptop rather than the cluster.

When you are done, remove the containers with docker rm -f plane node1 node2 relay and the network with docker network rm kube-air.

In your own KubeVirt

An Oberon machine also runs in plain KubeVirt, without Cozystack. It needs our virt-launcher image, which knows the RISC5 architecture, and KubeVirt’s ability to attach hooks to virtual machines switched on. How to do that is described in detail in the guide, which also has an example VirtualMachine. A cluster needs three such machines, a relay as an ordinary pod with a UDP service, and the commands at start in the machine’s annotation, which the hook passes on to QEMU. That is exactly what the Cozystack catalog does for you, so without Cozystack the easiest way is to see what its charts in the marketplace folder create.

In Cozystack

If you have Cozystack, plug in our catalog once with cozypkg:

cozypkg tap oci://ghcr.io/tym83/paleocomputing/machines:v0.1.21
cozypkg add paleocomputing.machines

The cluster administrator has to do two things for this: switch on the Sidecar feature gate in KubeVirt and install our virt-launcher image for your KubeVirt version. The details are on the project page.

After that, users see a Paleocomputing section in the dashboard, with OberonVM, OberonAir and OberonKube among other things. A Kube cluster is ordered with one form or one resource:

apiVersion: apps.cozystack.io/v1alpha1
kind: OberonKube
metadata:
  name: farm
spec:
  nodes: 3
  key: 00c0ffee00c0ffee
  deployments: web 4 Ticker; api 2 Ticker2

From this order the catalog creates an air, a control plane machine and three nodes, and the cluster forms by itself. Any machine’s screen opens with virtctl vnc under the tenant’s own rights, and the machine names are shown in the dashboard. The source of truth here is the form: its deployments are applied at every start of the control plane, so a changed form takes effect after a restart, and a change made over VNC lasts only until then. Deleting the order deletes all its parts too.

You can also build a cluster by hand from separate machines. Then create an OberonAir, and in each OberonVM name that air in the field air, the role in kubeRole (plane for the control plane and node for the nodes), the node name in kubeNode and the same key in kubeKey. The control plane’s deployments go in the field commands, and the cluster name, if it should not be kube, in kubeCluster. This is more fun if you want, for example, to put two clusters with different keys on one air and watch them not get in each other’s way.

What all this says about tomorrow’s infrastructure

At the start I mentioned that this is research as well, not just a joke, so let me try to put into words what the cluster on Wirth’s radio taught me. Not in the sense that everyone should rewrite Kubernetes in Oberon, but in the sense of which ideas are worth carrying from this small system into a large one.

State, not events. Inside its control plane Kubernetes has long worked on the level-triggered principle, but between components it still has events, watches, change streams and long-lived connections. Kube went further simply because the radio left it no choice: every message carries the full state, and the protocol needs no acknowledgements, no retries and no recovery after a broken connection. I think this approach applies far more widely than is usually assumed. Wherever the state can be described briefly enough and the network is unreliable, be it edge networks, satellite links, industrial networks or clusters of thousands of small devices, a protocol in which every message stands on its own turns out both simpler and more reliable.

Observability as a property of the protocol. Since the air carries everything the cluster knows about itself, I did not have to build observability. In a large system you cannot broadcast everything to everyone, of course, but the very thought that what should be observed is the messages between components, not the components’ reports about themselves, seems interesting. A check that looks at the behaviour of the nodes, not at what the control plane thinks of them, catches the control plane’s own bugs, and that is how nearly all of Kube’s bugs were found.

A unit of deployment the size of a module. An Oberon module with an interface key is very close to where WebAssembly is now heading with its component model: a small piece of code with an explicitly described interface, which can be loaded quickly and checked for compatibility before it runs. Oberon lacked isolation, and WebAssembly provides it without a separate process or kernel. Put the two together, and you get a pod that starts like a function call, is isolated like a container, and has its compatibility checked at load time, as in Oberon. I think tomorrow’s infrastructure may well be built on such principles.

Limits written in the code. All of Kube’s ceilings sit in two blocks of constants, and each has a reason. Kubernetes has limits too, but they are scattered across flags, documentation and operating experience, and many of them you learn about only when you hit them. I would like large systems to declare their limits and the reasons for them more honestly, rather than pretend there are none.

Simplicity that fits in a head. Kube can be read in an evening, and that is not decoration but a working property: when something went wrong, I found the cause by reading one procedure, not by going through issues in a tracker. Kubernetes will never be like that again, but new systems can be designed so that their core, the part responsible for the main idea, stays surveyable, with everything else built around it in layers you need not read.

Security in the very first version. The radio forced messages to be signed from the very start, and it cost about thirty lines. Systems that began with a trusted network and added protection later paid more for it. The lesson is trite, but Oberon illustrates it well: if the environment is hostile from day one, the right architecture comes about by itself.

And separately, about what not to do. Kube shows clearly that isolation, preemption and separation of rights are not superfluous complexity to be thrown out for simplicity’s sake, but what a shared computer is impossible without. This is exactly where Oberon lost, and any system that wants to be both simple and shared will have to find a way to get isolation without losing surveyability. It seems to me that today, for the first time, suitable building blocks for this exist, from WebAssembly to small verifiable kernels, and the only question is whether anyone will want to put together from them a system that can once again be read in full.

Instead of a conclusion

I set out wanting to check whether the idea of Kubernetes fits into a machine designed from start to finish by one person. It fits, together with rollouts, disaster recovery, a signed protocol and a store on disk, in about fourteen hundred lines of a language from the late eighties. Along the way it turned out that the machine’s limits did not only get in the way but also suggested solutions, and some of them proved better than the ones we are used to. It also became clear exactly where such systems hit their ceiling, and that is perhaps the most valuable part, because it explains why the world took another road.

Next on the list are a readiness check that the workload itself answers, so that a rollout waits until new code is really ready rather than merely started, and a way to apply deployments from outside the control plane without VNC. And perhaps isolation, at least partial, to see how much simplicity it would cost.

The project is open. The sources are in the repository; our code is under the Apache-2.0 licence, and the QEMU machine model, like QEMU itself, under the GPL. The cluster lab lives at tym83.github.io/paleocomputing/oberon/kube.html, and the other labs of the series and the page on the Cozystack catalog are on the project site. You can read about Cozystack itself at cozystack.io. I will be glad if someone switches the air off in the lab not for half a minute but in some cleverer way, and finds where Kube breaks. There have been many such findings in this story already, and each one taught something.

In the next part of the paleocomputing series we will build, from the specifications and the tender documents, Ada’s rivals: the programming languages that took part in the US Department of Defense competition but failed to win.

On 1 January 2024, Niklaus Wirth died — the man who gave us Pascal, Modula-2 and Oberon, won the Turing Award, and spent his whole life waging a stubborn war on bloated software. What fewer people know is that, already well into his seventies, he sat down and designed a processor of his own: small and simple, so that he could show students a whole computer at once, from the logic gates up to the windows on the screen.

In September 2026 I took that processor and tried to run it everywhere I could reach. First right inside a browser tab — and not as an emulator, but as the very circuit Wirth drew. Then in QEMU, as an ordinary virtual machine. Then in Kubernetes, which is normally home to a very different kind of VM, the sort that runs Ubuntu and databases. And finally in Cozystack, our cloud platform, where Wirth’s machine can now be installed with a single click from the catalog, the way you’d install PostgreSQL. Along the way I finally measured something programmers have argued about for decades — what array-bounds checking actually costs. I also ran a small language model on Wirth’s processor. And, since we’re being honest, with one careless move I sent every virtual machine in a live production cluster off on a migration (nobody was hurt, but it was an unpleasant few seconds).

This turned into a very long article, because a great deal happened over nine days, and a good part of it was my own mistakes, which I then had to find and fix. I’ve tried to write so that someone who has never heard of Oberon can follow along, and I’ve tucked everything only a specialist would care about under spoilers. You can read straight through, or jump to whatever part interests you. First comes the story of Oberon itself, then how my original plan fell apart, then the processor and bounds-checking, then the browser and the lab exercises, then QEMU, Kubernetes and Cozystack, and right at the end the language model and how to install all of this yourself.

All the source code, notes and instructions live in the repository github.com/tym83/paleocomputing, and the project site is tym83.github.io/paleocomputing. If you’d rather get your hands dirty first and read afterwards, open the lab: nothing to install, the machine boots right there in your browser.

What Oberon is

Oberon is at once a programming language and an operating system, built in the mid-1980s at ETH Zürich by Niklaus Wirth and Jürg Gutknecht. They started in the autumn of 1985, and by 1988 the system was genuinely up and running. By Wirth’s own account, two people wrote it in the hours left over from their day jobs — which, you have to admit, sounds fairly wild when you think about how many people it takes to write any operating system today. It ran on the Ceres workstation, also built at ETH, and students were taught on it until roughly the early 2000s.

The name, incidentally, came from Voyager. In January 1986 the probe sent back images of Uranus and its moons, and Wirth — who regarded Voyager as a model engineering project, a craft that kept working far beyond its design life — named the system after the moon Oberon. In the book he calls Oberon the largest moon of Uranus, though in fact Titania is bigger; even Wirth has his typos. And, for good measure, Oberon is also the king of the elves, which isn’t a bad namesake either.

The whole enterprise has its guiding idea set out in the preface to the project’s book: a system built from scratch should be something you can describe, explain and understand in full — one person should be able to read and understand the whole of it, from the processor to the windowing interface. By today’s standards that sounds almost like fantasy, because nobody alive understands a modern OS, together with its compiler and its processor, in full.

In 2013 Wirth released a new edition of the project, and in it he carried the idea all the way. The processor Ceres ran on had long since gone out of production, and Wirth decided that since everything else in the system was his own and comprehensible, the processor should be his own too. He designed a simple RISC processor — called RISC5 in the sources — and described it in Verilog, the language used to describe digital circuits. A circuit like that can be flashed onto an FPGA, a programmable chip, and out comes a real, working computer. Wirth’s ran on an inexpensive board with one megabyte of memory at 25 MHz. That’s roughly a hundred times slower than a single core of your phone, but it is more than enough for Oberon, because the compiler, as Wirth writes, compiles itself in about three seconds. The book and the sources for the system, the compiler and the processor are all in the open, and that is exactly what makes Oberon unique. One more detail that will matter to us: the system itself dates from the late 1980s, while the processor we’ll be running it on appeared only in the 2010s.

Why bother with a museum piece in 2026? I have three reasons. The first is how deeply Oberon shaped what we use today. Go — the language half of today’s cloud infrastructure is written in, Kubernetes included — names Oberon outright as one of its ancestors; it’s right there in the Go FAQ. And Robert Griesemer, one of Go’s three authors, did his doctorate under Wirth at ETH, and in his talk “The Evolution of Go” he draws a family tree with a straight line running from Oberon to Go. The second reason is that Oberon really does fit in your head. If you want to understand how a computer works from the gate to the window, I know of no better teaching aid. And the third reason is that in 1995 Wirth wrote “A Plea for Lean Software”, a manifesto against bloated software — the source of Wirth’s law, that software slows down faster than hardware speeds up, though Wirth honestly credited the observation to his colleague Martin Reiser — and Oberon was its proof that things could be otherwise. Thirty years on, matters have, to put it mildly, only got worse.

And Oberon, pleasingly, is still alive. It has active forks that were updated just days ago, and a Russian-speaking community, OberonCore, briskly discussing all of it.

Where this whole idea came from

I’d been keeping a list of forgotten systems for a long time. Operating systems, languages and whole machines that once worked, sometimes very well, and then vanished — often taking with them ideas nobody has properly reproduced since. The Burroughs B5000, which back in 1961 checked data types right there in the hardware. KeyKOS, for which a power cord yanked from the wall was a normal event, not a disaster. Transputers, Lilith, the iAPX 432. Articles and books have been written about all of them, but almost nobody actually runs them, and running them was exactly what I wanted — for real, with all the guts, to see what they can do.

That’s how the series I called Paleocomputing came about. The idea behind it is simple. Many of these systems lost not because they were bad but because they were expensive. Custom silicon for a custom language cost insane money, while ordinary Intel processors got cheaper faster than anyone could build something clever. The economics are different now. Programmable chips cost pennies, the open RISC-V architecture officially permits adding your own instructions to the processor, and the CHERI project — much more on it below — essentially brings back into modern processors the very hardware memory protection Burroughs was doing before I was born. Which means the old ideas can be tested by hand again, and that is what I set about doing.

For the first episode I chose Oberon, though I wanted Burroughs more. The reason is straightforward: Oberon has absolutely everything you need to work. The processor sources, the compiler, the operating system, the book that explains all of it, living emulators and ready-made disk images. You can dive straight to full depth instead of spending a month excavating documentation — and to start a series, scale was exactly what I needed.

How it was done

Let me admit up front that most of the code in this project was written not by me but by a neural network. More precisely, by several AI agents to whom I handed out different roles. Some wrote code, others reviewed it, and still others were given the job of finding out why the whole thing didn’t work — and took it very seriously. They’ll turn up throughout the text as reviewers and auditors, and I want it clear from the start that these are not living people but the same neural networks, which I set to play the part of a captious reader, a hardware specialist, or a measurement methodologist. They worked independently of one another and of the agent writing the code, and that, I think, turned out to be the single most useful invention of the whole project. What stayed with me were the design, the decisions, the endless questions about whether everything had really been checked, and a live cluster that, as it turns out, is very easy to upset.

I mention this not to ride a fashionable topic but because the rest of the story makes no sense without it. First, everything described here was done in nine days, from the 21st to the 29th of September, and that pace would have been impossible had I written it all by hand. Second, it’s the reason some of the code can’t be handed to the big open-source projects, which I’ll come to in the section on QEMU. And third, a neural network has one very characteristic habit: it loves to report that everything is done and all the checks are green. In practice that often meant the check simply wasn’t checking anything. So most of those nine days went not into writing code but into teaching the checks to blush honestly when something was broken — a thread that runs through the whole article.

The plan that fell apart in a day

On paper the plan sounded splendid. Wirth’s real processor was to run right inside a browser tab, cycle by cycle — and not as an emulator written from the documentation, but as his own circuit. A reader could change the instruction set of that processor right there on the page, press a button, and a few seconds later watch the operating system carry on running on the altered hardware. And on top of that I meant to lay three loud claims. That the whole computer, from gates to windows, runs in a browser. That if you add to the processor a special instruction which multiplies and adds in one go, a small neural network on it will run at least twice as fast, and you can do it in about a minute. And that if you add hardware array-bounds checking to the processor, you can for the first time honestly measure what such a check costs — because, as I confidently wrote in the draft, nobody has that number.

I gave myself two to four weeks. Right.

Before sitting down to write code, I sent the plan out to five reviewers, each with an area of their own: the schedule, the hardware, the compiler, the measurement methodology, and the browser. The brief was the same for all of them: find why this won’t work. The comments came to more than eighty kilobytes of text, and after them not much of the plan was left standing.

The most galling part was that the room for new instructions in the processor, which I thought I’d found, wasn’t free at all. I’d seen a bit in the instruction encoding that is always zero, and I was going to seat my new instructions there. The hardware reviewer opened Wirth’s circuit and showed that the processor simply doesn’t look at that bit — so on a real processor all my new instructions would silently execute as the most common instruction of all, a data move, and the program would quietly do something entirely unlike what was written. There was no free room in the instruction set at all. Later, it’s true, one small spot did turn up, where Wirth’s compiler always writes zeros, and there’s a separate story about how I tried to squeeze in there.

The two-times speed-up for the neural network didn’t survive the review either. The reviewers counted straight off the circuit how many cycles each operation takes, and even if the new instruction were entirely free, it could not in principle speed the program up by more than a factor of two — and the one actually built would be lucky to manage a few percent. And right there was the key point, later borne out: the bottleneck isn’t the missing instruction at all, it’s that Wirth’s multiplier is very slow, computing the product one bit per cycle.

The number nobody supposedly has proved quite an embarrassment. The methodologist brought a list of papers where the cost of hardware memory protection had been measured many times over, including on modern Arm processors and in the CHERI project. About the only thing that was relatively new was that I wanted to use, as the workload, a system that rebuilds itself — but even that is a detail rather than a discovery.

It also emerged that in Oberon you can’t turn off array-bounds checking in ordinary code at all; it’s welded into the compiler. There was simply nothing to compare Wirth’s system against without checks, and I had to plan from the very start for my own compiler patch that produces a check-free build.

After the review I rewrote the plan. Two claims remained in the first episode. The first, that the whole computer runs in a browser. The second, that you can honestly measure the cost of bounds-checking on a fully open system where everything is visible, from the processor circuit to the compiler. The neural network and its new instruction I moved to a separate episode. And a line I’m very fond of appeared in the risk table: that the number will most likely come out boring, around six percent — and that this is the result.

Getting ahead of myself: I did eventually run a neural network on Wirth’s processor, and the reviewers were right about the main thing — it all came down to the multiplier. But that’s nearer the end.

Running Wirth’s real processor

It all starts simply. Wirth described his processor in Verilog, and that description isn’t a program but a circuit — registers, wires, and the logic between them. To run a circuit like that without an actual chip, there’s a tool called Verilator, which turns it into a C++ program that faithfully recomputes every wire of the processor on every cycle. It runs slower than real hardware, but exactly like it, with no approximations. The processor still needs a screen, a disk, a keyboard and a mouse bolted on. I built that harness myself, reproducing precisely the interface the system expects, and I didn’t touch the processor itself by a single line. That was a point of principle for me, because everything I’m going to measure later has to be measured on his circuit, not on mine.

The first working day began at one in the morning on 22 September, and by lunchtime Wirth’s real circuit had booted the real Project Oberon. Booting takes about twelve million instructions, and on my laptop the simulation gets through it in four seconds. The picture at the top of the article is from this very run.

Getting it to run is only half the job; you have to be sure it ran correctly. For that there’s a technique called lockstep comparison. You take a reference you trust and drive it alongside your own machine through the same program, instruction by instruction, comparing the contents of every register after each one. If even a single bit diverges anywhere, you find out immediately, and you know exactly which instruction it was on. For the reference I took the emulator from the Norebo project, and across the entire system boot — nearly fifteen million instructions — my circuit never once diverged from it. There was a divergence at first, mind you, and it was my fault: on a reviewer’s advice I’d written a housekeeping value into memory just in case, one the emulator writes only in a particular situation that didn’t arise in this run. The right thing was to touch nothing.

Here I should say why this matters at all. Every cycle count in this article is computed by a separate fast model, because running the real circuit under heavy loads takes far too long. And that model can be trusted exactly as far as it matches the circuit. I checked the two against each other on the same boot and got a perfect match, down to the cycle — but at first the comparison claimed the model was off by nearly sixteen percent, and I very nearly believed it. The comparison itself was lying, because it was peeking at the instruction address one cycle earlier than it should. Had I published back then that the model was 16% off, it would have devalued every number in the project. Since then I have a rule: a negative result must be checked just as carefully as a positive one, because you can go wrong in both directions.

What array-bounds checking costs

Now for the central question of the first episode. If you write a[i] in C, and i happens to be larger than the array, the program will silently read or write someone else’s memory. From that simple mistake grew a huge share of the vulnerabilities of the last fifty years, from the buffer overflows of the nineties to today’s browser security bulletins. Guarding against it is very simple: before every array access, compare the index against the length and, if it’s out of bounds, stop the program. Many languages do exactly this; Oberon always does; and in C and C++ the check traditionally isn’t written, because it’s held to be slow. So I wanted to find out just how slow it actually is, on a system where absolutely everything is visible.

For that you need three variants of the same system. The first has no checks at all, and no such Oberon exists, so I made it myself by changing a single line in the compiler. The second has the software check, as Wirth’s does, where the compiler inserts a comparison and a jump to the fault handler before every access. And the third has a hardware check, where the processor gains a new instruction that does all of this itself in one cycle. That instruction gets a section of its own. For the workload I took the Oberon compiler compiling several modules of the system itself, because it’s a real, large program rather than a synthetic test.

The first number I got was under half a percent, and I was delighted that checks cost almost nothing. Then I realized I’d compared the wrong things. I was running two different compilers, one with checks and one without, and the check-free compiler simply did less work, because it didn’t have to insert checks into anyone else’s code. What you must compare is the same work — that is, build one and the same compiler in two variants and give them the same task. Once I fixed that, the number grew several-fold.

There were a few more passes after that, and the honest answer was this. As long as the checks are removed only from the compiler itself, they cost about two percent of the time. And if you remove them from the whole system the compiler runs on, including its handling of text, files and memory, it comes to almost five percent. So for every hundred cycles of the compiler’s work, between two and five go towards making sure the program never runs off the end of an array. Whether that’s a lot or a little, each reader can decide; to my taste it’s very cheap for the absence of an entire class of vulnerabilities. With one caveat the reviewers made on the very first day: I have only one workload, so this is a number about the Oberon compiler, not about every program in the world.

Here the first surprise was waiting for me. When I recounted which checks actually sit in the code, I found there are very few array-bounds checks in Oberon, and the overwhelming majority are checks that a pointer isn’t empty — that it isn’t NIL. In the compiler alone there are more than three hundred of these against a handful of index checks. So the cost of safety in Oberon comes mostly from the program being unable to dereference a null pointer, and arrays as such are a secondary matter here.

How I taught the processor to check for itself

Now the hardware check. The idea is simple. Instead of two instructions — a comparison and a conditional jump — the processor gains one new instruction, which I named CHK. It takes an index and an array length and, if the index is out of range, jumps to the fault handler itself. All of it in one cycle instead of two.

The difficulty turned out to be where in the instruction to put the array length. An instruction on Wirth’s processor is 32 bits, and almost all of them are already taken. The only place the processor genuinely doesn’t use — which I proved by enumerating every possible value — is twelve bits in the middle of the instruction. Twelve bits means arrays of up to four thousand elements, which in Oberon is the overwhelming majority, so I was pleased and put the length there.

And I got a very nasty result. When a check fires in Oberon, the system reports where and exactly what error occurred, and it takes the error number from the very same instruction bits where I’d put the length. I ran an experiment with a real error, a program that reaches past the end of a hundred-element array. The ordinary system honestly reported that the index was out of the array’s bounds. Mine, with the new instruction, reported just as confidently that there had been a null-pointer dereference, because a chunk of the number 100 had landed in the error-number field. A programmer getting that message would go off hunting a non-existent pointer bug and lose half a day to it. That’s worse than the system saying nothing at all.

After that I tried to wriggle out of it a couple of times by shrinking the length to eight bits, and both times it went wrong. First the statistics showed eight bits were enough for most arrays, but then it emerged that the majority in those statistics were identical buffers for file names, hardly ever used, while the hot loops that run millions of times work with precisely the large arrays and don’t fit in eight bits. The solution turned up elsewhere. The new instruction, as it happens, doesn’t need a destination register, because it computes nothing and only checks, and the four bits that usually name that register are free. The length can be sliced in two: the four high bits go there, and the eight low bits into the middle of the instruction, positioned so they don’t touch the error number. That gave both the twelve bits of length and the correct error messages.

`ifdef WITH_CHK
assign CHK     = ~p & ~q & ~u & v & (op == 1);
assign chkFail = CHK & (B >= chkLim);
`else
assign CHK     = 1'b0;
assign chkFail = 1'b0;
`endif

So how much does the hardware check save in the end? On the compiler it removes about a sixth of the cost of checks; on a mixed load with arrays of various sizes, a tenth; and on purely computational code with small arrays, a half. A half, by the way, is the ceiling, and it will never be more, because two instructions became one. The difference between the loads has a very simple explanation. The instruction helps only where the array length fits in twelve bits. Where the array is larger, the compiler falls back on the old software check, and there’s no gain.

And one more lovely story from this section. For the second workload I wrote a small program with a sort and a matrix multiply. In the checked variant it fell over immediately with an out-of-bounds message, and the bug was in my program itself: I’d computed an index wrong and was overrunning the array by a factor of three and a half. The check-free variant ran silently, as though all were well, and quietly wrote two and a half thousand numbers into someone else’s memory. The bug was found only because the checks were on, and honestly I can’t think of a better advertisement for bounds-checking.

How much the new instruction costs in hardware — how much room it takes on the die, and whether it slows the processor down — I measured too, and the answer came out boring. It adds less than a percent of logic and doesn’t touch the clock frequency at all. The interesting part here is how I arrived at that boring answer, because along the way the noise in such measurements proved larger than the effect itself.

The auditors didn’t accept the work

By the evening of the first day I had a lot of pretty numbers, and I handed them to five auditors for acceptance. At 17:40 they reported back, and all five wrote the same word: NOT ACCEPTED, in capitals.

The most unpleasant and, at the same time, the most useful was the mutation audit. Its idea is simple and very cruel. The auditor deliberately introduces plausible errors into the processor — changing a “greater than or equal” to a “greater than” in a comparison, say — and watches whether my tests notice. If the tests stay green on a broken processor, they aren’t checking anything. Of thirty errors introduced, my tests caught ten. Two thirds of the broken processors passed every check as sound.

The reasons were very instructive. In one place a check that had failed to run counted as passed, and the report proudly announced that three of five cases had been checked and no errors found. In another, the test build was configured so that its failure didn’t stop the run, and success was determined by a smiley on the last line of output, so zero tests executed gave a fully green report. And an instruction that supposedly checked the system boot in fact checked nothing at all: with a processor that had one of its instructions broken, the machine hung right at the start of the boot, and the check still reported success. On top of that, the system boot contains not a single operation on fractional numbers, so my claim that the comparison had checked all of the processor’s arithmetic was simply untrue.

The auditors’ main conclusion was that almost all the errors happened in carrying the results over into the text. A number obtained under one set of conditions migrated into the summary as a general one. We rewrote everything we found, added the same kind of deliberate breakage to the automatic check on every change, and introduced a rule that any check must be able to fail, and that this must be demonstrated. And the next day one more gem surfaced. In all two hundred-plus processor tests, the very first instruction of the program wasn’t actually executing, because of that same bus-during-reset bug I wrote about above. It was impossible to notice, because every test began with an instruction whose result didn’t matter anyway. The most galling part is that I’d already found and fixed this bug on another rig, and had even left a comment there about tripping on exactly this.

And at 19:30 on the first day the circle closed. The Oberon compiler, running on Wirth’s real circuit, compiled itself, and the result matched byte for byte what the emulator produces. The next day the same thing came off inside the system itself, with windows and a mouse. A script of clicks and keystrokes made the system rebuild itself completely, and every file came out exactly as in the original disk image. Along the way it emerged that the official 2016 system image isn’t entirely consistent with itself: a couple of its modules were out of date, and one was missing altogether. The ordinary life of any living project — and honestly, it rather moved me.

Oberon in a browser tab

Now the first claim of this episode, the one about the browser. Wirth’s circuit, which Verilator turned into a C++ program, can then be compiled further into WebAssembly — the format in which a browser can run ordinary compiled code at nearly native speed. And then what’s spinning in the tab isn’t an emulator someone wrote from the documentation, but the very same circuit Wirth flashed onto his FPGA, with all its wires and cycles. I’ll note again that the screen, disk, keyboard and mouse around the processor are mine, built to the same interface as Wirth’s, and that the video controller, which on a real machine takes a little of the processor’s time, doesn’t figure in the measurements.

At first I was told this was a bad idea, because recomputing every wire is too slow for a browser. In practice the browser version runs almost as fast as the same simulation launched directly on the computer — a couple of percent apart. That’s about six times slower than Wirth’s real machine, so the system takes a few seconds to boot, and after that you can quite happily work with it. And the whole machine together with its disk image weighs about three hundred kilobytes — less than the average picture on any news site.

Thanks to this, the machine can be dropped into any page with two lines:

<script type="module" src="https://tym83.github.io/paleocomputing/oberon/embed.js"></script>
<oberon-machine base="https://tym83.github.io/paleocomputing/oberon/"></oberon-machine>

Details and settings are on the Embed it page. There you can also switch the machine to the processor with the new check instruction and place your own files on its disk before boot.

The page where you change the processor

Originally I wanted the reader to be able to change the processor’s instruction set themselves, right there on the page. That, I’ll freely admit, I didn’t do, because rebuilding the circuit from Verilog directly in the browser proved too heavy. The fallback idea from the same plan worked instead: build several processor variants in advance and let you switch between them. On the Change the processor page there are two cores — the ordinary one, like Wirth’s, and one with the new CHK instruction — and on each you can run the same loop that walks an array.

On the ordinary processor, one array access in this loop takes eleven cycles, two of which go on the software check. On the processor with the new instruction it’s ten, because the check takes one cycle instead of two. One cycle out of eleven, exactly as intended. That boring line from the risk table came true.

And the main thing this page checks isn’t even the cost of the check. The ordinary system, with no new instructions at all, must boot on both processors in exactly the same way, with the same image on the screen and the same number of instructions executed. If, after the new instruction is added, the old programs behave even slightly differently, then I’ve broken compatibility and ended up with some other machine.

Thirteen lab exercises

Since the machine runs in the browser, you can learn on it. The lab has thirteen exercises, and each is checked by the machine rather than taken on trust. The check looks into the memory, registers or disk of the emulated computer itself and sees whether you did what was asked. The exercises are graded by level: first just look, then change, break, measure and, finally, build your own.

And this is what the Oberon interface looks like in action. A middle-click on the text System.ShowModules opens the list of loaded modules, and another, on Hilbert.Draw, draws a Hilbert curve. No buttons, only text:

By default the lab opens in English, and switches to Russian with a button or a link carrying ?lang=ru, and the choice is remembered. Beside it sits an eight-chapter handbook that begins with what’s real here and ends with what we measured, and it has an English version too. If you teach computer architecture and want to take these exercises for your own use, write to me and I’ll help you set them up. They’re open, and the machine can be embedded in your own page with the same tag.

A separate embarrassment of this section is that the lab on the site was dead for a while. The exercise list was empty and an eternal spinner hung on the screen. First I found one error in the page’s code and fixed it, but that didn’t help. The real cause was that the page’s file had no closing tag on its script. I’d considered this harmless — the browser will forgive it — and had even taught the site’s automatic check to forgive it too. But by the HTML standard, if a file ends inside an unclosed script, the browser marks that script as already executed and simply doesn’t run it, issuing neither an error nor a warning. Now the automatic check opens the site in a real browser and requires the page to have as many exercises as the sources do. In short, I no longer say the browser will forgive anything.

And what does the check cost on modern processors?

All right — on Wirth’s processor the software check costs two cycles out of eleven, and the hardware one costs a single cycle. But Wirth’s processor is a very simple machine that executes one instruction after another and guesses at nothing. How do things stand on the processors in our laptops and servers?

I took the same array loop, wrote it in C and in Rust in three variants. In the first there’s no check at all; in the second it’s written the way the language itself writes it; and in the third the compiler is forbidden to throw it away. All three variants I ran everywhere I could reach. Let me note straight away, because without it the numbers can’t be read: the loop is deliberately chosen to be as favorable to the check as possible. The array is small and always sits in the processor’s fastest memory, and the check always passes, so it’s easy for the processor to predict its outcome. On top of that, I forbade the compiler to vectorize — to process several array elements with one instruction — even though in real code checks are costly primarily because they get in the way of exactly that. I wanted to see the check on its own, without everything else.

First, the check itself hasn’t changed at all in forty years — everywhere it’s the same comparison and conditional jump as on Wirth’s machine. Second, the check as the language writes it cost nothing at all anywhere, because modern C and Rust compilers worked out for themselves that the index in this loop never goes out of bounds and threw the check away. Wirth’s compiler can’t do that; it always leaves the check in.

And why is the check invisible on AMD and Apple but visible on the server Arm? This strikes me as the most interesting result of the first half of the project. A modern processor executes several instructions per cycle and reorders them itself to keep all its execution units busy. If a program has little work, the extra check instructions simply drop into the free slots and cost nothing, like a passenger who boards a half-empty bus and crowds no one. I checked this by gradually adding work and checks to the loop and watching for when the checks start to cost time. On AMD one or two checks really do cost nothing, and from the fourth they begin to. On the server Arm each check adds a little time from the very start, because that core has fewer free slots. Wirth’s processor has no free slots at all — it executes one instruction at a time, and every check instruction is a cycle that always gets paid for.

What CHERI does

CHERI is an architecture in which a pointer knows its own bounds — that is, along with the address it stores where the region of memory it’s allowed to reach begins and ends, and the processor checks those bounds itself on every memory access. Such a pointer is twice as wide as an ordinary address, and a program can’t forge it. I wanted to put CHERI on the same ladder, and not on a big, complex processor where an extra instruction can cost anything at all, but on the closest relative to Wirth’s processor. I took CHERIoT-Ibex, a simple 32-bit core that also executes instructions one at a time and has no cache, and ran the same loop on its circuit.

The result was very telling. On CHERI the check takes not a single instruction in the loop, because it’s built into the memory read itself. The loop with protection is exactly the same length as the loop without it. To make sure the protection really works, I narrowed the pointer’s bounds to 32 elements, and the program fell over on precisely the 33rd. And there is no check-free variant on CHERI, and never will be, because there’s simply nothing to turn it off with. The published work confirms this. On CHERI the check itself is nearly free, and what you pay for is the pointer width, because wide pointers take up twice the room in memory and in cache.

And so a ladder emerged. On Wirth’s processor the software check costs two instructions and two cycles, the new CHK instruction costs one instruction and one cycle, and CHERI costs zero instructions. The wide modern processors stand off to the side: they have as many instructions as Wirth’s, but as long as the core has free slots those instructions cost almost nothing. And between CHK and CHERI a gap yawned on this ladder for a long time — one I closed only at the very end of these nine days.

Wirth’s machine in QEMU

The browser is wonderful, but I wanted Wirth’s machine to live where ordinary virtual machines live, with its own console, disks, restarts and all the trappings of a proper hypervisor. Almost every VM on Linux is launched, one way or another, by QEMU, a program that can impersonate computers of the most varied architectures. Wirth’s processor, of course, isn’t among them, so it had to be written from scratch — that is, QEMU had to be taught to understand all of RISC5’s instructions and its idiosyncratic fractional arithmetic, and then a board had to be assembled around the processor, with memory, a disk, a keyboard, a mouse and a screen.

We had one advantage here that almost nobody writing a new architecture for QEMU has. Usually the correctness of such work is checked against documentation and test suites, whereas we had the processor’s own circuit and a reference emulator already checked against it over fifteen million instructions. So we checked QEMU not against paper but against the circuit, instruction by instruction, by that same lockstep technique. Across one and a half million instructions of the boot, no divergence was found; the image on the screen matched the circuit down to the last dot; and the list of modules that opens on a click in System.ShowModules was exactly the same, with the same addresses in memory. The number of dark dots on the screen after boot — 18,607 — became my reference from then on, and wherever the machine ran, that is what I compared the screen against.

You can build and run it like this (in detail, in the QEMU guide):

git clone https://github.com/tym83/paleocomputing && cd paleocomputing
make -C qemu build        # builds QEMU in a Docker container
docker create --name oberon-payload ghcr.io/tym83/paleocomputing/oberon-run:v0.1.17
docker cp oberon-payload:/opt/oberon/payload/prom.bin .
docker cp oberon-payload:/opt/oberon/payload/oberon.dsk .
docker rm oberon-payload
.qemu-work/build/qemu-system-risc5 -machine oberon -bios prom.bin \
  -drive if=none,id=sd0,file=oberon.dsk,format=raw -vnc :0

The first command builds QEMU with our processor; the next three pull the bootloader and the system disk out of a ready-made image; and the last one starts the machine. Watch the screen with any VNC client at 127.0.0.1:5900, and if you add chk=on to -machine oberon, you get the processor with the hardware check. Booting without hardware acceleration takes up to a minute, so don’t be alarmed if at first you see only the bootloader screen.

Why this is a separate build rather than a patch to mainline QEMU is also worth explaining. The QEMU and libvirt projects don’t accept code that a language model had a hand in writing — even where it’s only suspected. I have a public repository that says honestly how it was made, so this can go upstream only if someone rewrites the code by hand. The GPL, meanwhile, expressly permits your own build, and the hardest part of the work — an exact description of the processor’s behavior and a way to check any implementation of it — remains useful regardless. One further nuisance is that our processor is written against the very newest development version of QEMU, so already-released versions won’t build it.

libvirt, which won’t take your word for it

The next layer is libvirt. It’s the library through which almost everything that manages VMs on Linux talks to QEMU, from the virsh command-line tool to KubeVirt, which is coming up next. The question was whether a new architecture could be plugged in with no edits at all. It couldn’t — but not for the reason I expected.

First I simply put the architecture risc5 in the machine description, and libvirt refused right away, saying it didn’t know such an architecture. So I got clever, called myself an architecture it did know, and slipped it my own program. libvirt refused again, but this time more deeply. It doesn’t trust what’s written in the machine description; it asks QEMU itself what it can impersonate, QEMU honestly calls itself risc5, and there’s no such name in libvirt’s list. Deceiving it through the machine description is impossible.

So there’s no getting around a libvirt edit, and the edit was tiny — about ten lines in five places. I made it so that the list of architectures is taken from a separate file, and the next old machine would be a new line in that file rather than new code. The build checks each edit separately, because an edit that silently failed to apply is worse than none: libvirt would build without errors but not know the architecture. This came in handy at once, when in a new version of libvirt the relevant piece of code moved to another file, and the build announced it loudly instead of quietly building a broken library.

And here at once is a fine story about why you can’t trust your eyes. The first screenshot from under libvirt looked entirely correct, but a byte-by-byte comparison with the reference found several thousand differences. I managed to suspect the mouse, the moment the screenshot was taken, and libvirt itself — and I’d simply miscomputed the address of video memory from hexadecimal into decimal and missed by 512 bytes. That’s exactly four rows of the screen, and the difference is completely invisible to the eye. After the fix, the screen matched the reference byte for byte.

KubeVirt without a fork

One layer up sits KubeVirt. It lets you run virtual machines in Kubernetes the same way you run ordinary containers, and manage them with the same tools. Inside, each such VM has a housekeeping pod, and it’s there that libvirt and QEMU live. I needed to slip a machine of an entirely different architecture in there, and to do it without making my own copy of either KubeVirt or Cozystack. A private copy of a large project stays with you forever, and you have to drag it by hand through every update, so I very much wanted to avoid that.

A standard extension point that few people know about came to the rescue. Before starting a VM, KubeVirt can hand its description to an external handler and then run whatever the handler returns. The handler can be an ordinary script sitting in the Kubernetes configuration, so you don’t even need to build a separate image for it. Our handler receives the description of a perfectly ordinary VM — with an Intel processor, disks and networking — and reshapes it into Wirth’s machine. It changes the architecture, turns off hardware acceleration (which, for a foreign architecture, doesn’t exist anyway), substitutes our emulator, bootloader and disk image, and throws out everything Wirth’s machine doesn’t have — which is almost everything KubeVirt adds by default.

The emulator itself and the patched libvirt do have to be placed into the housekeeping image that KubeVirt starts all its VMs from. Everything else KubeVirt and Cozystack handle themselves. So for Wirth’s machine the cluster needs exactly two things an ordinary user can’t do: allow external handlers, and install our housekeeping image. The administrator agrees to this once, and after that any user installs the machine from the catalog, much as you’d install a driver package.

There was one more quiet trap with the housekeeping image. We take KubeVirt’s standard image and replace only libvirt in it, and the version of our libvirt has to match, exactly, the one already in the image. Get it wrong and the image builds without a single error, but our library ends up beside the standard one rather than in place of it, and it’s the standard one that runs. So now the build checks that there’s exactly one libvirt in the image, and the correspondence between KubeVirt and libvirt versions is recorded in a single file and can’t be set by hand.

Sixteen seconds to a cluster-wide migration

This is probably the most instructive story of the project, and it’s about my own inattention; the code has nothing to do with it.

The housekeeping image KubeVirt starts VMs from is set for the entire cluster at once. To swap it out, I edited the KubeVirt configuration right on the live cluster — our working rig, on which, among other things, other people’s VMs were running. Sixteen seconds later KubeVirt began migrating every virtual machine in the cluster onto the new image, without stopping them. The thing is, Cozystack has automatic updates of running machines enabled by default, and KubeVirt did exactly what it was told: since the housekeeping image had changed, all the machines needed moving to the new one. Over five hours it attempted to migrate the machines more than a hundred times, and more than half the attempts failed. Nothing crashed and no data was lost, and the successful migrations incidentally showed that our image can migrate machines — but the picture was not a pretty one. Here’s how I stopped it:

kubectl -n cozy-kubevirt patch kubevirt kubevirt --type=merge \
  -p '{"spec":{"workloadUpdateStrategy":{"workloadUpdateMethods":[]}}}'

This command turns off automatic updates of machines, but it’s an emergency brake, not a solution. It doesn’t cancel migrations already under way, the setting will come back at the next Cozystack update, and going entirely without auto-updates is also bad, because after a KubeVirt update the machines would stay on the old housekeeping image. The most galling part of this story is that I knew where to look. I simply checked that the new image existed, and didn’t think to check what it would do to the cluster.

Why half the migrations failed became clear from the logs, and the image had nothing to do with it. By default KubeVirt migrates no more than two machines at a time off a single server; the rest wait their turn and often don’t get one. Quotas got in the way too. During a migration a machine exists in two copies at once and takes up twice the memory, so a machine that has eaten its user’s entire quota will never migrate live. The quota needs headroom, at least enough for the largest machine.

A catalog for Cozystack

Cozystack is an open platform from which you assemble your own cloud on top of Kubernetes. We at Ænix build it together with the community, and it’s part of the CNCF, the foundation where Kubernetes itself lives. The platform has a web interface, users — here called tenants (like separate accounts in a cloud) — virtual machines, and an application catalog holding databases, Kubernetes clusters, caches and the rest. And recently a mechanism for pluggable catalogs appeared in the community. Anyone can publish their own set of applications and plug it into their platform, and then those applications show up for users alongside the built-in ones. For paleocomputing I made exactly such a catalog, inventing nothing beyond what’s already in the platform.

The catalog is split into four parts, because they have to be trusted differently. The first holds the machines and environments for users — the real VM with Wirth’s processor, the browser lab, the handbook, and a bundle that installs all of it at once. The second holds the Oberon language environment, in which you can run Oberon programs as ordinary jobs in the cluster. The third places boot images into the platform’s shared storage, and the fourth swaps out that KubeVirt housekeeping image. These last two touch the whole cluster, and so are installed only with the administrator’s explicit consent.

The catalog is plugged in like this:

cozypkg tap oci://ghcr.io/tym83/paleocomputing/machines:v0.1.17
cozypkg tap oci://ghcr.io/tym83/paleocomputing/languages:v0.1.17
cozypkg add paleocomputing.machines
cozypkg add paleocomputing.languages
# what touches the whole cluster is installed only with the administrator's explicit consent
cozypkg tap oci://ghcr.io/tym83/paleocomputing/images:v0.1.17
cozypkg tap oci://ghcr.io/tym83/paleocomputing/platform:v0.1.17
cozypkg add paleocomputing.platform --allow-privileged

The tap command only plugs the catalog into the platform; add installs its contents. That distinction once confused me too, and at first it wasn’t in the documentation at all.

Before installing the last part, be sure to look at the automatic-machine-update setting in KubeVirt. With Cozystack’s default settings, changing the housekeeping image will send every VM in the cluster off on a migration, and this happens on install, on uninstall, and at every KubeVirt update. Exactly what happened to me. This hole was found by one of the reviewers while checking this very article, so now, if auto-update is on, the component changes nothing and waits until the administrator explicitly permits the migration. You can permit it with the setting allowWorkloadUpdate: true at install time, or with the annotation paleocomputing.io/allow-workload-update=true on the KubeVirt resource. And one more thing. The catalog-plugging utility doesn’t yet verify the digital signature, so if you’re installing this somewhere more serious than a home rig, check the signature yourself; the instructions say how.

After it’s plugged in, a Paleocomputing section of its own appears in users’ catalogs. There’s a wrinkle, though, which I hit while taking the screenshots for this article. The current Cozystack web interface shows only three sections in its sidebar — the ones hard-wired into its code — and hides all the rest at the very end of the full list of applications. At first I couldn’t find my own section either. And yet the whole point of a pluggable catalog is to bring your own sections, not to dissolve into someone else’s. So we’ll be fixing this in Cozystack itself, and the interface will have to show any sections a catalog brings.

The machine is installed with a form in the web interface or with one short description:

apiVersion: apps.cozystack.io/v1alpha1
kind: OberonVM
metadata:
  name: wirth
spec:
  memory: 128Mi
  hardware: chk     # or base, the ordinary Wirth processor

There’s no screen for the machine in the web interface yet; in the current versions only ordinary VMs have one. You can look at it with the virtctl utility, with the user’s own privileges and any VNC client:

virtctl -n tenant-sandbox vnc oberon-vm-oberon-vm-habr

It’s arranged so that the next old machine won’t require new code. Everything that sets Wirth’s machine apart from the others is recorded in one small file, which I call the machine’s passport: the architecture, the emulator, the bootloader, the disk, the processor variants and the memory limits. Everything else is shared, and the handler for KubeVirt is one and the same for all machines — it simply reads the passport. The next machine, Lilith for instance, is a new passport, not a copy of the code. To prove this isn’t empty talk, the tests include a second, fictional machine of a different architecture, and it builds without a single edit to the code.

kind: Machine
name: oberon
image: ghcr.io/tym83/paleocomputing/oberon-run:v0.1.17
domain:
  arch: risc5
  machine: oberon
  emulator: /usr/local/bin/qemu-system-risc5
  vcpus: 1
  graphics: vnc
  terminationGracePeriodSeconds: 0
payload:
  path: /payload
  files:
    - {name: prom.bin,   role: firmware, qemu: ["-bios", "{path}"]}
    - {name: oberon.dsk, role: disk,     qemu: ["-drive", "if=none,id=sd0,file={path},format=raw"]}
variants: {base: {}, chk: {chk: "on"}}
memory: {default: 128Mi, min: 128Mi, max: 1Gi}

Everything green, and nothing works

While I was fussing with the cluster, the same trap recurred six times in one evening, and I started a note under exactly that title. Each time a check reported success while in fact nothing worked. The catalog is built, but the main file isn’t inside it. The install succeeded, but the machine isn’t running. The server answers that all is well, but the image doesn’t contain the machine itself. The build is green, but the file being copied doesn’t exist. Then there was a seventh time — those very sixteen seconds.

On the evening of 27 September there was an episode I’m still a little ashamed of. We were reworking how the machine is described in the cluster, and the releases came in a queue. The first stopped at its own checks. The second hung dead, because the task that prepares the machine’s disk was waiting for the machine to start, and the machine was waiting for the disk. The third found two more bugs. At that point I lost my patience and asked the agent why we were shipping release after release, and couldn’t it just check things properly the first time (in the original this was put more forcefully).

It could, and after that the release process changed. Now every change is first built as a test version, a separate sandbox in the cluster switches to it, and a scenario, acting as an ordinary user, installs the machine, checks that it started on the right processor, compares the screen against the reference, restarts it, deletes it, and confirms that nothing was left behind. Only if all of that passes does the change reach the main branch and become a release. A second such scenario checks the component that changes the housekeeping image, and, for good measure, runs an ordinary Ubuntu on our image to confirm we haven’t broken the neighbors. The first release to pass all of this before publication was v0.1.14, and the stuck machines came up by themselves afterwards.

Amusingly, the scenario itself had bugs of its own, even while the system was working. For example, Ubuntu’s boot was at first verified via a helper program inside the guest, which simply isn’t in a clean Ubuntu image, so the check would have waited forever. Then it was verified via the login prompt on the console, and a console without a real terminal stays silent. And the wait function counted only the pauses, not the total elapsed time, so twenty minutes of waiting turned into two hours. The script waited very patiently by a long-since-booted Ubuntu.

The component that watches the housekeeping image

Swapping out KubeVirt’s housekeeping image by hand is a bad idea, for three reasons. It’s shared by all the cluster’s VMs, so a mistake breaks everyone. It has to match the KubeVirt version, and after a Cozystack update it will stay old and break all the machines. And my first manual method incidentally froze several other KubeVirt settings I had no intention of touching.

So the catalog has a separate component that looks at the KubeVirt configuration every half a minute and puts its own part in order. If we have an image for the current KubeVirt version, it installs it. If not, it removes its edit, and the cluster returns to the standard image. Wirth’s machines then won’t start, but everything else will work. If anything at all looks questionable, it likewise removes its edit, and if it couldn’t read the configuration, it touches nothing. It makes its edit in such a way that if KubeVirt ever changes the shape of its configuration, the edit won’t slot the image into the wrong place but will loudly refuse to apply.

And this component the sandbox caught too. On the live cluster it crashed, because it passed one of the utilities information about every server in the cluster at once, and on a real cluster that information is enormous. In the tests the servers were tiny and the test cluster consisted of a single one, so nothing crashed there. Now a test with a cluster of three thousand large servers turns red on the old code.

Servers on Intel and on Arm

When I started building all this for Arm servers too, I was asked why builds for different processors are needed at all if we’ve already added the Oberon architecture to KubeVirt. It’s a good question, and the confusion is natural, because there are two architectures here. One is the architecture of Wirth’s machine, which QEMU impersonates, and it doesn’t depend on the server. The other is the server’s real processor, Intel or Arm. The emulator and libvirt are ordinary programs, and they have to be built separately for each server processor. It’s like an emulator for a games console, where the console is one thing but the emulator’s builds for Windows and for Mac are different.

Our housekeeping image was built only for Intel. And since it replaces the standard image for every VM in the cluster, on a cluster of Arm servers not a single machine would start — not just Oberon. The Arm build went through at once, but the actual launch on Arm stopped three times. First, the image with the bootloader and disk existed only for Intel. Then, that one of the build tools is also released only for Intel. And the third problem was the most interesting. On Arm, KubeVirt always gives a VM modern firmware with its own flash memory, and Wirth’s machine has no flash memory at all, so QEMU exited immediately after starting. On Intel this isn’t visible, because the default firmware there is different. Not one of these three problems was visible in the code or in the built image — only on a live server with the right processor.

Now all of this works on two versions of KubeVirt and on both processor types, and the screen matches the reference in all four combinations. In fairness I’ll note that on Arm I tested bare KubeVirt in a throwaway test cluster, and I haven’t yet run Cozystack on Arm servers.

# 1. allow external handlers (on servers without /dev/kvm you also need useEmulation: true)
kubectl -n kubevirt get kubevirt kubevirt -o json \
  | jq '.spec.configuration.developerConfiguration.featureGates |= ((. // []) + ["Sidecar"] | unique)' \
  | kubectl replace -f -

# 2. install the housekeeping image for your KubeVirt version
git clone --depth 1 -b v0.1.17 https://github.com/tym83/paleocomputing && cd paleocomputing
KV=$(kubectl -n kubevirt get kubevirt kubevirt -o jsonpath='{.status.observedKubeVirtVersion}')
echo "$KV ghcr.io/tym83/paleocomputing/virt-launcher:$KV-paleo-v0.1.17" > /tmp/launchers.txt
kubectl -n kubevirt create configmap kubevirt-paleo-launcher-status
export KUBECTL=kubectl KUBEVIRT_NAMESPACE=kubevirt LAUNCHER_TABLE=/tmp/launchers.txt
R=marketplace/repos/platform/packages/system/kubevirt-paleo-launcher/files/reconcile.sh
until sh $R once && [ "$(kubectl -n kubevirt get cm kubevirt-paleo-launcher-status \
  -o jsonpath='{.data.state}')" = Applied ]; do sleep 10; done

# 3. install the machine with ordinary Helm and open its screen
python3 marketplace/tools/pin-images.py --release v0.1.17
helm install wirth marketplace/repos/machines/packages/apps/oberon-vm \
  -n oberon --create-namespace --set storageClass=<your StorageClass>
virtctl -n oberon vnc oberon-vm-wirth

The second episode: a language model on Wirth’s processor

Remember the claim from that very first plan, that a new instruction which multiplies and adds in one go would double the speed of a neural network on Wirth’s processor? The reviewers demolished it, but a line with their estimates stayed in my list of deferred tasks afterwards. The new instruction, by their reckoning, would give a few percent; a fast multiplier, more than one and a half times. It sounded like a result, but behind those numbers there was neither a program nor a way to reproduce them — only arithmetic off the cycle table. I’d wanted to check it by hand from the start.

The task boiled down to this. A small language model was to run inside the Oberon system itself. A program in Oberon, compiled by the system’s own compiler, reads the model’s weights from a file and prints text, and all of it executes on Wirth’s real circuit, cycle by cycle. And then I’d need to work out where the time goes and check both of the reviewers’ estimates.

Wirth’s machine hasn’t much memory — one megabyte for everything, screen included — so the model came out truly tiny. It predicts the next letter from the previous eight, and has about forty-three thousand parameters. For comparison, the models we chat with have millions of times more. Yet even a model like this takes up almost half of the machine’s free memory. I trained it on an ordinary laptop in a few dozen seconds, on the text of Alice’s Adventures in Wonderland, long since in the public domain.

Here’s what the model writes if you start it with alice was:

alice was one thought all the tell you spo

Shakespeare can rest easy. But what interests us here isn’t the literature, it’s the stopwatch.

The main requirement was that the text the model prints on Wirth’s circuit match, byte for byte, a reference computed on an ordinary computer. For that, the reference had to be computed not with the usual Python facilities but in exactly the same arithmetic as Wirth’s processor, repeating every operation in the same order. Otherwise nothing would have matched. I separately worked out how far Wirth’s arithmetic diverges from the standard, and found that almost all the intermediate numbers differ in their last digits. The generated text never once diverged, mind you, but that’s simple luck. A comparison against ordinary Python would have been right almost always, and one day inexplicably wrong — the worst kind of bug, because it can be neither reproduced nor explained.

The text matched everywhere. On the emulator, where this check now runs automatically on every change; on Wirth’s circuit; on the circuit with the fast multiplier, of which more below; and inside the real system with windows. The program and the weights are placed on the disk, two middle-clicks first compile the program and then start the generation, and the finished text is read off the disk and matches the reference. In QEMU, the same.

Where the time goes

On Wirth’s processor with its native multiplier, the model prints about nine letters a second. On a live machine, where the video controller takes a little of the processor’s time, slightly fewer. In short, nine letters a second, with luck.

When I looked at what the processor was busy with all that time, the picture was very clear. Almost forty percent of all cycles go on fractional multiplication, and most of that time is simply waiting for the slow multiplier to finish counting. Wirth’s is serial and computes the product one bit per cycle, so a single multiplication takes twenty-six cycles. Another quarter of the time goes on reading from memory. And here another interesting thing emerged. The simplest line of the program, where a product is added to a sum, Wirth’s compiler turns into twenty-seven instructions, of which only four are useful. All the rest is reading and writing the sum to memory, the loop counter and — yes — bounds checks and NIL checks again. The thing is, Wirth’s compiler can’t keep variables in the processor’s registers and goes to memory for them every time. One of the reviewers predicted this from the sources on the first day and guessed the cycle count almost exactly. This, by the way, isn’t an oversight but deliberate simplicity: the whole of Wirth’s compiler is under three thousand lines and builds itself in seconds, and complex optimization simply didn’t fit that budget.

The fast multiplier

Since it all comes down to the multiplier, I wrote a fast one. It does the same thing as Wirth’s, but not one bit per cycle — all at once, in one or two cycles — with the rounding and other subtleties rewritten exactly after Wirth. It’s switched on the same way as the CHK instruction, with a single toggle when building the processor.

The main requirement of it was as strict as everything else: the results must match Wirth’s multiplier to the last bit. I ran tens of millions of pairs of numbers through it, including all the awkward edge cases, and found not a single divergence. And to make sure the check can notice errors at all, I deliberately spoiled the rounding in one place, and it found tens of thousands of divergences.

With the fast multiplier the model ran 1.6 times faster — about fourteen letters a second instead of nine. The estimate from the profile predicted exactly that, but on a simple machine with no caches that’s more a confirmation that the profile was computed correctly than a real prediction. Even if multiplication became entirely free, the program couldn’t be sped up by more than 1.64 times, because the other cycles don’t go anywhere. So the reviewers’ number didn’t quite come true, but on the main point they were right: the multiplier was indeed the bottleneck.

For the system’s ordinary work, though, the fast multiplier gives nothing at all. The compiler building itself does, across forty million instructions, a mere thirty-one fractional multiplications, and the system boot does none. The slow multiplier was a perfectly sensible choice for a machine on which you write text and build programs. The language model was the first task for which it became the bottleneck.

And what of the new instruction that multiplies and adds, the one this all began with? I didn’t implement it; I estimated it from the counters. On Wirth’s ordinary multiplier it would give one and a half to six percent, on the fast one up to ten. So the main lever here isn’t the new instruction but a compiler that learns to keep the sum in a register instead of running to memory for it. But that I no longer measured.

In hardware the fast multiplier was even smaller than the native one, because an FPGA has ready-made hardware multiply blocks (its DSP slices), and all the serial-counting logic simply vanished. The single-cycle variant, mind you, noticeably lowers the processor’s top frequency, while the two-cycle one is almost as fast and doesn’t touch the frequency. The native 25 MHz holds with a large margin in both cases, so you’d only have to choose between them if someone wanted to overclock Wirth’s machine.

Two finds along the way

The first find concerns the comparison of fractional numbers. Wirth’s compiler compares them the same way as integers, through subtraction and a check of the processor’s flags, and for some comparisons it uses the overflow flag. Only that flag is changed exclusively by integer operations. As a result, if an integer overflowed somewhere in the program before a comparison of fractional numbers, the comparison can give a wrong answer. I checked this with a very simple program. First it honestly says that one is less than two; then it overflows an integer; after which it reports, just as confidently, that one is not less than two. This reproduces both on the emulator and on the circuit. My model has no overflows, and the reference checks for that. I claim no novelty here: surely someone has seen this already; correct me if you know where.

The second find was funnier. The model needed to save a text file to disk, and here it emerged that in the whole project nobody had ever written anything to our QEMU, because the system boot only reads from disk. And it couldn’t write, because the disk was attached without write permission, and the very first attempt to save anything brought QEMU down entirely. On top of that, Oberon’s filesystem places new data past the end of the disk, and the disk image was exactly the size the system occupied, so the file silently wasn’t saved either.

When I fixed that and ran the machine in Kubernetes, writing still didn’t work — now because of file permissions. The task that places the machine’s disk onto its volume was supposed to permit writing, but did it in a way that actually permitted nothing. All that time the disk in the cluster was read-only, and had anyone saved a file inside Oberon, the machine would have crashed. Nobody noticed, because nobody was saving anything. Now the permissions are set explicitly, including on already-created disks, and the sandbox check scenario asks QEMU itself whether it opened the disk for writing.

All the details, tables and commands for reproducing this are in the episode write-up, and the essentials can be reproduced in a couple of minutes with cd impl && make lm-check && make lm-profile.

Descriptors, between CHK and CHERI

The CHK instruction checks an index against a length that the compiler baked into the instruction itself, which means the compiler has to know the array length in advance. CHERI keeps the bounds in the pointer, and the check can be neither forgotten nor bypassed. Between these two extremes there historically sat descriptors as well, as in that very Burroughs B5000 of the early sixties. A descriptor is a pointer that carries the array length with it, and the processor checks it on every access, so the compiler no longer needs to know anything in advance. I wanted to put this rung on the ladder and measure it on the same fully open system.

The very first observation narrowed the task sharply. In Oberon the compiler knows the length of almost any array in advance, because the language simply has no pointers to arrays and no variable-length arrays. There’s one exception: when an array is passed to a procedure prepared to accept an array of any length. Inside such a procedure the length isn’t known in advance, and that’s exactly where a descriptor has something to do. Everywhere else CHK already copes.

I fit the descriptor into an ordinary 32-bit word. Wirth’s processor has a 24-bit address, but the machine has only a megabyte of memory, and twenty bits are enough for a megabyte, so I gave the remaining twelve bits to the array length — up to four thousand elements and a bit. And with it one new instruction, which takes a descriptor and an index, computes the element’s address and, if the index runs past the length, jumps to the fault handler. An ordinary address, fed in place of a descriptor, looks like a zero-length array, so any access through it is immediately treated as an error. This is the closest analogue of CHERI’s central rule — that without permission there is no access — that you can build without a special hardware tag.

The new instruction was checked as strictly as CHK: with the same deliberate breakages, lockstep comparison against the emulator, and a check that the ordinary system boots the same on a processor with the new instruction as on the ordinary one. And once again the first attempt lied. The run with deliberate breakages first reported that it had caught them all. The script that introduced the breakages was itself crashing, the circuit wasn’t building, and the fact that nothing had built was being counted as a caught breakage. Now a circuit that fails to build counts as a failed check, not a passed one.

On the same loop as on the two-core page, the descriptor turned out to be the fastest of all — faster even than the variant with no checks at all, eight cycles per access against nine. There’s no miracle in this. The new instruction also computes the array element’s address, which usually takes two more instructions, and had I made an identical instruction without the check, it would run just as fast. The lesson here is a different one, and I like it: if the bound travels with the pointer, the check can be hidden inside an instruction you need anyway. That is exactly how it’s done in CHERI, where the check sits inside the memory read.

Next I taught the compiler to use descriptors where the length isn’t known in advance, and asked the system to rebuild itself with the new compiler. It rebuilt, and the next generation of the compiler matched the previous one byte for byte. This is an important check, because if the compiler had anywhere forgotten to turn a descriptor back into an ordinary address, the address would have been wrong and there’d have been no match.

And then the real system found two boundaries I’d never have thought of myself. The first concerns array length. In the whole of Project Oberon there isn’t a single array longer than four thousand elements, but in the tools I built it with there turned up a sixteen-thousand-element buffer, and the compiler honestly refused to build it rather than silently truncating the length. The second boundary was more interesting. When I built many modules in one run, memory in the emulator crept past a megabyte, and my descriptor can address only a megabyte, so the build simply hung, silently. Hence a very telling conclusion. A pointer with bounds can’t be squeezed into the width of an ordinary address, because as soon as there’s more memory, there’s no room left for the length. That’s exactly why CHERI’s pointers are twice as wide as an address and, on top of that, compress the bounds cleverly.

What does all this construction cost on large loads? On a synthetic program that works heavily with arrays of unknown length, descriptors give a speed-up of about a quarter, for the same reason as in the loop: they save on computing the address. On the compiler, though — a real program — they instead give a tiny slowdown, under a percent, and I spent quite a while hunting for where it comes from, because by calculation it should have been a speed-up. It had nothing to do with descriptors: every time the program passes a string to a procedure, the compiler now assembles a descriptor for it, which made the code a little longer, and the system spends slightly more time loading modules. Had the compiler prepared the string descriptors in advance, this wouldn’t happen, but that I no longer did.

On the FPGA the processor with descriptors still holds its 25 MHz, has about five percent more logic, and the new instruction landed, for the first time, on the circuit’s critical path — its longest signal path — meaning it could limit the frequency in future. CHK never once did that.

In the end the ladder looks like this:

A descriptor removes the check from the loop just as CHERI does, but in Oberon it has almost nothing to protect beyond what CHK already does. And the key thing that sets it apart from CHERI: a descriptor stays an ordinary number, and a program permitted low-level operations can assemble any descriptor from any length and any address. It’s precisely here that CHERI puts a special tag that can’t be forged. A system built entirely with descriptors I haven’t yet booted on the real circuit; this processor isn’t in the browser or the catalog yet either; and all the details and commands for reproducing it are in the episode write-up.

How to install all of this yourself

There are four ways, from the very simplest, where you need install nothing, to your own cloud.

The easiest is to open the lab in your browser and do at least the first four exercises. You’ll see how an interface where any text can be a command works, write your first module, and break the machine by writing garbage straight into screen memory. If you don’t want the exercises, the Just run the system page holds the bare system, and the Change the processor page lets you see the cost of a check with your own eyes. The middle mouse button is stood in for by a click with Alt, and the two-button chord by a click with Shift. Of the things worth trying yourself, I’d suggest a middle-click on System.ShowModules, to see how few modules the system needs after boot, and on Hilbert.Draw, because it’s simply pretty. And then lab exercises twelve and thirteen, the most hardware-facing: in one you measure the three processor variants yourself, in the other you add a new built-in procedure to the language and rebuild the compiler right inside the system.

If you want Wirth’s machine as an ordinary VM, build your own QEMU from the commands in the QEMU section or from the guide. It’s worth trying the processor variants with the hardware check and with descriptors there.

If you have your own KubeVirt, there’s a guide and the short version in the spoiler above. The two latest KubeVirt versions are supported, and servers on both Intel and Arm. You can start with a throwaway test cluster; the same scenario the automatic check runs can be launched by hand. And I’ll repeat the warning once more, because I got burned by it myself: the housekeeping image is changed for all of the cluster’s VMs at once, so on a working cluster, first check whether automatic machine updates are on.

And if you have Cozystack, it’s enough to plug in the catalog with the commands from the Cozystack section, and your users will get Wirth’s machine, the lab, the handbook and a bundle that installs it all at once. The details are on the project page. If you need to show students or colleagues how a whole computer is built, this is probably the quickest way. As for the automatic machine updates, the component now asks about them itself and won’t touch anything without explicit consent.

And if you’d like to check up on us, the command cd impl && make deps && make check will, in about eight minutes, run all the processor tests, the system boot, the lockstep comparison against the reference, the compiler self-build, and a rebuild of the whole system with a byte-for-byte comparison. For this you’ll need Verilator and a C++ compiler.

What I took away from this

First, on bounds-checking. It’s cheaper than commonly thought, but I wouldn’t call it entirely free. On Wirth’s processor — very simple, no caches, no branch prediction — it cost the compiler between two and five percent of the time, depending on which part of the system you remove it from. And most of that cost comes from null-pointer checks, not array-bounds checks. On large modern processors, in my deliberately favorable loop, it’s invisible, because the extra instructions drop into free slots; on the server Arm it’s visible, if only a little. In real programs checks are costly primarily because they stop the compiler from processing arrays in batches, and that I didn’t measure. CHERI hides the check right inside the memory read, and then there’s no check in the program at all.

Second, the bottleneck often turns out not to be where you look for it. For the neural network everyone, myself included, wanted a clever new instruction, while the bottleneck was the old slow multiplier. Replacing it sped the model up 1.6 times, whereas the new instruction, by estimate, would have given a few percent. The reviewers said this on the very first day, and a measurement a week later confirmed it. First measure, and only then add hardware.

Third, a pointer that knows its own bounds has to be made wider than an ordinary address. My descriptors, fit into an ordinary word, ran up against a megabyte of memory and arrays of up to four thousand elements, and that is exactly why CHERI’s pointers are twice as wide.

Fourth, the nastiest bugs are found only by a live environment. My tests were good, but the most interesting things were found by real KubeVirt, real Cozystack and a live Arm server — from a machine that needs no power management, to a network layer that silently doesn’t answer, to sixteen seconds before the whole cluster migrated. Since then I release a new version only if it has first passed a full check in the sandbox.

And fifth, a neural network writes code fast and proves it works slowly. Nine days for all of this happened only because the code was written by agents. But more than half of those days went into teaching the checks to blush honestly, and for every pretty number there was an auditor — also an agent — who took it apart. If a check has never once turned red, it most likely isn’t checking anything, especially when you badly want everything to work out. And a separate open question is what to do with code like this in open-source projects. QEMU and libvirt won’t accept it at all; the CNCF, by contrast, takes it calmly; and how open source is to live with this isn’t yet very clear.

In place of a conclusion

All of this is open. The sources are in the repository, our code under the Apache-2.0 licence, and the processor for QEMU under the GPL, like QEMU itself. The site with the lab is tym83.github.io/paleocomputing. Every find, with the commands to reproduce it, is in the repository in the impl/docs folder, and about Cozystack itself you can read at cozystack.io.

Coming up next in the series: the Burroughs B5000 with its hardware memory protection; Lilith, Wirth’s very first machine, which will need only a new passport; and, patience permitting, my own board with Oberon on an FPGA, to finally measure the cycles not in simulation but on real hardware.

If you use Oberon in teaching, or simply wrote in it once, tell me in the comments — I’m very curious where it lives now. And do correct me if I’ve gone wrong somewhere; judging by this article, that happens to me regularly.

A quick poll: what will you do after this article?

• Open the lab and break the machine by writing garbage into screen memory
• Go check auto-updates for the VMs on my own cluster. Right now
• Rewrite everything in Oberon (Go is basically Oberon, only with goroutines)
• Keep turning off bounds-checking for the sake of a couple of percent
• I wrote in it back in the nineties, and I have something to say in the comments
• Made it as far as the poll, which is an achievement in itself

P.S. A special thank-you to Niklaus Wirth. I never met him, but for nine days I talked to his circuit, and it lied to me almost never. Unlike my own checks.

Spoiler: full autonomy didn’t happen, and I ended up wrestling with the project quite a bit. But the process pulled me in completely, and the end result beat every expectation I had.

People in the community kept asking how it actually went. Fair question — building this taught me a great deal about driving models effectively, and it turned up a pile of working methods and patterns that have since made me much more productive in everyday work too.

What Blockstor is

I called the project Blockstor. It’s a block device orchestrator. Roughly: you request a replicated volume of the size you need, and that volume gets created on several nodes in ZFS or LVM and set up for replication with DRBD — a smart, network-aware take on RAID 1. The system supports snapshots, replica reallocation, resize, automatic failover, and more.

Blockstor is not a storage system designed and written on a blank page. I built the experiment on LINSTOR — a mature distributed block storage manager that I’ve run in production myself for years.

LINSTOR was close to an ideal storage system for me, because its API types are Kubernetes-like. Its backend logic, though, is organized in a rather peculiar way. The main problem, as I see it, is the request-based model: for most API calls it goes out to the nodes in real time and polls their current state to build a response. To my mind that’s an unacceptable pattern for a distributed system, because it doesn’t hold up at scale. The absence of a reconciliation loop also makes automatic recovery hard. I’m not going to second-guess the developers’ decisions — I’m sure they had their reasons. But given how committed I am to Kubernetes patterns, I asked myself how I — an experienced architect and developer — would write a system like this in Kubernetes-native logic.

Rewrites from one language to another aren’t rare, by the way. That, for example, is how the rusternetes project came about: Kubernetes being rewritten from Go to Rust. But how is that even possible? Kubernetes has an enormous codebase with an enormous number of person-hours in it. How do you convince yourself that the resulting “AI slop” works correctly? And can you rely on it at all, let alone run it in production?

TDD

The answer lies in how Kubernetes itself is developed in the open. Free, community-driven projects have a body of established practice, and contributors do their best to follow it.

All kinds of people contribute to projects like these, at every skill level. That makes tests critical: tests are what keep functionality someone contributed from breaking later. Say you land a new feature, it passes every test, and it goes into the project. Later someone lands another feature. If a test for your feature fails while theirs is being checked, that means the new functionality has a problem that breaks yours — and the system automatically refuses to let the change through. Over time, tests in public projects have become so important that they’re now valued more highly than the code itself. Tests like these are exactly what makes rewriting X in Y possible.

The rusternetes project claims:

Actively conformance-tested against the official Kubernetes e2e test suite — currently passing 94% of conformance tests (415/441) across 160 rounds of testing.

So passing the tests is the primary yardstick for whether a project counts as a Kubernetes replacement.

When I see a bird that walks like a duck and swims like a duck and quacks like a duck, I call that bird a duck.

— The famous duck test

Hooks and setup

For the same reason, the first thing I did was force the model into test-driven development (TDD), where tests are written before the implementation. Tests let you pin down the conformance requirements you need. They also give me confidence that new functionality arriving during development won’t break what already works.

On a colleague’s advice I wired in golangci-lint right away and made it mandatory for the model to run all code through it. @lexfrei — the colleague in question — argues that this step saves a substantial number of tokens.

The exit gate and a deterministic result

This is probably the first and most important lesson: you can tell a model “keep writing until the tests pass”, and that’s a good condition on the artifact it produces. That’s exactly the method I used to build this project. With one catch: I had no tests, because LINSTOR doesn’t publish a test suite for its own functionality.

All I had to go on was years of running LINSTOR, dozens of my own talks and articles, and the Apache 2.0 code of projects in the LINSTOR ecosystem — golinstor, the Container Storage Interface (CSI) driver, Piraeus-operator, and their API contracts. LINSTOR itself is licensed under GPL, which rules out using its code to build an Apache 2.0 project on top of it. That was the central challenge, and it’s precisely why I went the clean-room route.

So my main job became defining those exit gates. Get them right and I don’t have to police the whole resulting codebase: if the code satisfies the conditions I set myself, that is the evidence the job is done.

The API as a contract

I like the type definitions LINSTOR uses in its API. On top of that, both the official CSI driver and Piraeus-operator work with those types, and I had no wish to rewrite either of them.

From experience, designing APIs is one of the most painful jobs in this profession. You can always change backend logic. Ship one breaking change to the API and everything falls apart on the client side. So an API should change rarely, in exceptional cases only — and the risk of getting it wrong at the very start is high. That’s why I decided to use exactly the types the original project uses, and map them onto Kubernetes-first CustomResourceDefinition (CRD) types.

While building Cozystack and selling solutions based on it, we’ve run into the same question from prospects more than once: what’s the story with your API? What if we put you into production, build a solution around you, and then your API changes and we have to rewrite our half of the project?

As a reminder: LINSTOR’s code is published under GPLv3, and I planned to publish Blockstor under Apache 2.0. Reusing the original code was off the table. So the first task was to find compatible contracts in the Kubernetes tooling around it: the golinstor library, linstor-csi, and Piraeus-operator, all distributed under Apache 2.0, plus the official LINSTOR documentation, licensed under CC BY-SA.

That gave me my first contract: the API must be compatible with the LINSTOR Go library, linstor-csi, and Piraeus-operator.

The test environment

The model needed somewhere to work and somewhere to test what it produced. For the test environment I picked a beefy bare-metal node and a generated test suite that brought up a Kubernetes cluster on Talos inside virtual machines (VMs) and deployed Blockstor into the cluster. That choice was deliberate: Blockstor needs the DRBD module, and DRBD has a habit of hanging when it’s configured wrong, so I needed a fast way to stand up and recreate a broken environment. Blockstor’s architecture also stores configuration as Kubernetes CRDs, so using Talos closed the question of bootstrapping Kubernetes itself.

The first result

When the first proof of concept (PoC) was ready, the model had built me a working prototype, guided only by the sources above and its own dataset. For the interaction model, though, it had implemented the very same request-based model as the original LINSTOR — presumably picked up from the project’s documentation. And it already worked! I could talk to the API using the official CLI, though there were hundreds of bugs and gaps.

To fix that, I asked the model to freeze the API contracts as tests and rewrite the logic on controller-runtime, the way I needed it. With a clear, deterministic goal, the architecture that came back began to resemble what I actually wanted: fully asynchronous, with the API translator as a separate pluggable module.

Later I had the model implement a set of end-to-end (e2e) tests that requested volumes through the official CSI plugin and the Kubernetes kubernetes-csi/csi-test framework. In other words, the model kept working until Blockstor provisioned volumes and snapshots through standard Kubernetes abstractions. After a while, I had a working prototype. The system was still a long way from stable, though, which left me with a question: how do you reach the stability I needed when there are no tests to start from?

This is where all my material went in — my articles on debugging LINSTOR, my talks, my Claude Code debugging skills, our plunger scripts integrated into Cozystack, and bugs reported on GitHub. I made the model study all of it and compile a set of issues that had to be tested and that our system had to satisfy. Out came a huge Markdown file. I reviewed it and sent the model off to test and fix every problem it had caught, on the test rig. There were many such iterations, and each one took a colossal amount of time. Most of the time the model would bring up the environment, run tests, fix errors, re-provision the environment — and all of that took time.

Eventually I started asking how to speed the process up.

Speeding up development

That’s where the idea of parallelizing agents came from. Some problems could be closed out at the unit-test level. Others could only be verified on a real environment. Running the same set of bugs through again and again, I arrived at this method:

1. I ask one agent to gather the references and put together a plan of sorts for the bugs.
2. Once I have the huge document, I ask the model to dispatch a batch of agents, each working a specific bug.
3. In the code, every agent gets its own isolated environment and its own worktree, where it submits its fixes.
4. Each agent’s job is to study the problem, bring up an environment, prepare a fix and tests, and hand all of it back to the main agent.
5. The main agent pulls the work from every agent into the main tree — and that repeats for several iterations.
6. In the end, the number of agents running at once reached 60: 30 at the unit-test level, 30 on e2e in a concrete environment.

After a while the project started to look like something you could actually use. Plenty of things still weren’t stable.

The marathon continues

While nursing this whole zoo along, I was already doing integration testing myself and checking the results by hand. I’d ask Claude for access to an environment, drive the linstor CLI manually, and try to reproduce the bugs, of which there were still plenty.

At that point I had to stop building large blocks and start digging into far more meticulous testing of the user journey. The bulk of the problems went away once I made the agents implement tests using the official CLI and laid out the user journey for Day-2 operations from the official LINSTOR documentation. In some places, though, the model started spinning its wheels: over all that time it never got every test passing reliably, and I had to step in personally. Drawing on what I know, I asked it to walk me through each problem, then put a set of leading questions to it about the architecture.

The fine-tuning stage

A lot of my questions came down to the asynchronous nature of the controllers. DRBD is by nature fussy about when, how, and at what point configuration gets applied, so it was critical to build a stable state machine — one that would let the reconciliation loop through to a specific action only when the conditions were met.

The main problem here was snapshot logic. To create a snapshot, LINSTOR first freezes I/O at the DRBD level, then sends the command simultaneously to the several nodes holding the backing device in ZFS. Once the operation succeeds, LINSTOR unfreezes DRBD. All of it has to happen instantly, and on error the state has to roll back automatically so that the container or VM never blocks while working with its data.

The second problem was how to skip the initial sync of a new replica. I knew the observable behavior from years of operating LINSTOR: the first replica records its current generation identifier, and later replicas take that as their starting value and sync from there.

First I tried to reconstruct the exact command sequence from observed behavior alone, running operations against a live controller and capturing the commands. But the mechanics aren’t documented publicly, and observation on its own didn’t add up. So I fell back on the clean-room technique. One agent — the “dirty room” — reconstructed a functional specification from the original sources: which commands run, and under what conditions. It recorded those functional facts and nothing else, carrying over no code and no expression of it. The second agent — the “clean room” — had no access to anyone else’s sources in the first place, and implemented the approach from scratch, strictly from the specification.

One more problem was consistent node-id allocation — every DRBD replica needs a unique number in the cluster, from 1 to 8 — along with allocating TCP ports for replication, which in the current implementation is no longer tied to DRBD and is handed out from a per-node pool instead. This is exactly where the state machine above paid off.

Building the CI system

By now it was clear the experiment was reaching its final stage, and I started thinking about the project’s future. To finish it and get to production, we needed a serious continuous integration (CI) system that would guarantee no untested code in the codebase. Given the volume of tests, we couldn’t let a run stretch over several hours, so they had to be parallelized the same way.

And we built one: for every pull request it ran a big pile of tests across six or seven runners in parallel and returned “ok” or “not ok”.

I worked in that mode for several more rounds, until CI was genuinely green and stable. Then we switched from local worktrees to pull requests on GitHub.

The final stage

Data loss is not acceptable, so before declaring the project finished I had to check thoroughly that Blockstor behaves correctly and reliably on that front. Reading and reviewing all the generated code was beyond both my energy and my capacity. On the other hand, Blockstor — like LINSTOR — is essentially an orchestrator. The data itself is stored by ZFS and DRBD, and I had no doubts about their reliability. What mattered was confirming that the controller really does configure resources correctly and survives dropouts.

But how do you establish whether the current code can be trusted with production? It’s an ambiguous and difficult question. And how do you get a deterministic answer to it? Here I used one more pattern I worked out over the course of the project.

My colleague @lexfrei had mentioned earlier that models are terrified of being told their actions could lead to serious financial losses. I decided to use that as an exit gate. I launched an agent responsible for releasing the code to production and had it work under conditions where its life and financial well-being “depended” on the final result. It drew up an acceptance plan with a long list of items, including a 24-hour burn-run on real infrastructure. A second agent tried to satisfy those requirements. That went on for several more days and several releases, until the agent in charge of shipping to production finally gave its blessing.

A system nobody uses can’t be called stable, though. So the next step was integrating Blockstor with Cozystack. Our test suite covers many functions at once: requesting volumes with and without DRBD, RWX, snapshots, and other details. The new agent’s job was to prepare a draft pull request, turn the tests green, and get through the final stage.

As of July 15, 2026, that pull request isn’t merged yet — but there’s a good chance you’ll soon have a new Kubernetes-native storage backend in Cozystack. Watch this space.

Editor’s note (September 2026): Blockstor is still a separate experimental project — github.com/cozystack/blockstor, Apache 2.0 — and it does not ship in Cozystack. LINSTOR/DRBD via Piraeus remains the storage Cozystack ships and its default backend. Blockstor sits on the roadmap as an opt-in backend for 2027.

The main takeaway

Blockstor still has experimental status. But the experience let me speed up and automate work on other projects considerably, and I now apply it every day.

Plenty of people still think of AI as an autocomplete tool. For me that stopped being true a long time ago. AI-first development isn’t occasionally asking a model to write a function. It’s rebuilding your entire engineering process around agents, context, tests, skills, plans, review, and automation.

Working that way, you can take on tasks that used to look too big for a small team — experimenting with a Kubernetes-native implementation of a LINSTOR-class storage system, for instance. But it only works on one condition: you need discipline. Without a plan, AI creates chaos far faster than a human can. With a plan, tests, agents, and proper review, AI becomes a force multiplier for engineering.

And this, I think, is what building complex infrastructure looks like from here on: not one engineer against an enormous codebase, but an engineer as the architect of a process, with a swarm of specialized agents working around them. The engineer’s job isn’t to watch each agent. It’s to build a system where the agents watch themselves and come to a human only with what genuinely needs one.

Join the community

• Blockstor on GitHub
• Cozystack on GitHub
• Telegram group
• Slack group (Get invite at https://slack.kubernetes.io)
• Community Meeting Calendar

What dozens of AI agents taught me: how I wrote the Blockstor storage system as an experiment was originally published in Ænix on Medium, where people are continuing the conversation by highlighting and responding to this story.

This article is the story of how we fixed that with aeman — a tool we built ourselves and recently open sourced. But I have to start further back.

Where this came from

Years ago I worked at another company, and it had two internal systems with wonderful names: Ford and Nixon. Colleagues there wrote about them at length, so here’s the short version. They’re boards for tracking daily tasks, where an engineer sees exactly what they’re working on today. Not a three-month backlog, not a hundred kanban columns — your day, and nothing else. For years that system kept a lot of remote teams working effectively and kept many customers’ infrastructure running.

Let me be honest: the process behind aeman isn’t entirely my idea. I took a working process-management system I’d seen at my previous company and mapped it onto our own processes and tasks. Our board now forces us to focus on what matters most — the business tasks.

Unlike Trello, this model starts from the assumption that one person can work on tasks in several teams during a single day. And the daily plan must not turn into a wall of tasks. If an engineer has 6 to 10 tasks planned but can realistically finish only 3 or 4, they end up with a permanent sense of debt, while management gets a false picture of heavy utilization. People also start switching between tasks constantly, and the brain naturally reaches for the ones that are easier to close. Hard tasks can then sit untouched for a long time. So the daily plan has to be short and honest.

Why not use an off-the-shelf tool?

I tried to build a process like this on existing tools: Trello, Asana, Notion, GitHub Projects — with varying success. Notion came closest, but I wasn’t willing to drag the whole team into yet another system and pay for it for the sake of the boards alone. And over time the board inevitably grew until it no longer fit on a screen.

GitHub Projects turned out to be the most viable option. It’s free, it offers a convenient API, and it already has everything you need: sprints, priorities, statuses, and other entities. It also extends well. That’s why we worked on it for a long time.

But it became clear fairly quickly that the problem wasn’t the tool’s capabilities — it was the UX. It takes far too many extra actions to create a task, assign an owner, and fill in the required fields. And we never managed to build a comfortable daily process around GitHub Projects. Sprints exist, for example, but they proved too awkward to use: you can’t close the current sprint and start the next one with the unfinished tasks carried over in a single action, or quickly push a task to “later” so it drops out of sight and stops competing for an engineer’s attention.

Another problem showed up over time. At every sync we were effectively looking at one column — In Progress. That’s where all the work happened, while cards in the other columns slowly turned into a task graveyard: still on the board, but no longer part of the daily process and going nowhere. GitHub Projects turned out to be an excellent place to store tasks and a weak one for managing a team’s daily focus.

That’s exactly why, when I started building aeman, I chose GitHub Projects as the backend of the new system and built the user interface and the workflow on top of it.

The core idea: exactly the cards you need today

The goal of aeman is to surface the tasks that matter today and focus the engineer on them. In an ideal world, an engineer leaves the morning sync with a list of tasks they will definitely work on that day. Everything else moves to tomorrow or to next week — and physically disappears from the board so it stops catching your eye.

The second idea: unplanned work has to be visible. Everyone has days that get eaten by a sudden incident or an urgent request from a neighboring team. Usually that work is recorded nowhere, and at the end of the sprint nobody can explain where the time went. In aeman it has a zone of its own: you log it after the fact as a single card, and the day stops vanishing without a trace (screenshots in the sections below). At the daily sync you can then talk it through and move the task into the planned block.

The third idea: multi-team work is a property of the model, not a filter. In a small company one engineer can work on tasks in several teams — this is especially true of founders, who still juggle everything. In aeman, teams are a dimension of the board: a sprint is tracked per team, each team has its own weekly plan, and the Me board collects one day’s cards from all of the engineer’s teams at once. You don’t have to hop between boards to see your day. A task also moves freely between people and teams — it can change assignees from one day to the next, and that’s a normal mode of work the tool shouldn’t get in the way of.

And the fourth idea: a team lead needs a tool that shows the status of every task within a minute, without pulling people away with extra questions.

Philosophically, aeman is closer to Todoist than to a classic kanban board. Cards are deliberately short: in normal mode you see only the title, and the description opens on a double click. The description takes free-form links — pull requests, GitHub issues, Telegram chats, anything else — and aeman recognizes them automatically and shows a button for jumping straight there. That keeps the main screen clean, and closing another task delivers the satisfying feeling familiar to anyone who uses a good task manager.

What it looks like

The Me board: your day

An engineer opens aeman in the morning and sees their day: cards grouped into four colored zones. This isn’t the Eisenhower matrix; the breakdown is different:

• Red — urgent, has to be done today
• Gray — planned, ordinary planned work, which may take more than one day
• Yellow — unplanned, whatever landed during the day
• Green — nice to have, do it only once everything else is finished

The board pulls in cards from all of the engineer’s teams, meaning from the current sprint of each of them. The team selector at the top narrows the list to a single team, or hides every card that’s done, blocked, or in review, leaving only what you can work on right here and right now, when it’s time to get into deep work. Each card has a completion slider from 0 to 100% in steps of 10% (dragging the progress bar to done is surprisingly satisfying), a stage (Review / Locked / Recurrent / Done) that recolors the bar, a counter of days in progress, and links to an issue or a PR. On the right there’s a notes panel: a personal log of the day where you can write things down as you work. That way you don’t have to dig through your memory at the next standup — you skim the notes and report what you did. Notes live in the context of the day; tomorrow the board is clean, but you can always step back a day and reread them.

The Team board: the whole team at a glance

The team lead’s view: a grid of people by zones for a chosen day. Columns are engineers with their avatars, rows are the same colored zones. You see immediately who’s working on what, whose work is on fire, and who’s overloaded. The lead creates cards from here too: a couple of clicks and a task is created, assigned, and prioritized.

In Me mode the team lead can also impersonate an engineer (the View as button) — look at the board through their eyes and tidy it up if needed.

Daily sprints and carry-over

Our sprints are short — one day — and they count forward, not backward. At the morning sync the team first discusses what got done yesterday, and then the team lead hits Carry over: every unfinished card moves into the new sprint (today), and finished ones stay in yesterday’s history. Recurring tasks are recreated automatically. That’s what day planning is here, and it takes minutes.

If planning decides a task definitely isn’t happening today, there are +1 day and +1 week buttons: the card disappears from the board until its day, and no history is lost. When that day comes, it shows up again.

The weekly plan

Here we departed from the original idea and started fitting our own workflow onto the tool we’d ended up with.

The founders and I used to plan the team’s week by writing a new list of weekly tasks into Google Sheets and sending each team a “focus for the week” message on Slack. That process has now moved into aeman as well.

Under the team grid lives the weekly plan: the team’s business tasks for the week, laid out in two lanes, “by Wednesday” and “by Friday”. Once a week the founders put tasks there, and the tasks wait their turn. The team lead drags a plan card onto an engineer; it appears on that engineer’s daily board while staying in the plan with a colored marker. A shared progress bar shows how the team is doing against the week, and the weekly Carry over moves whatever is still open into the next week.

Reviews, subtasks, and the activity log

Sending a task to review creates a linked card for the reviewer, with a feedback loop: while the review is open, the original sits at the Review stage, and once the reviewer takes their own card to 100%, the original is released automatically. Big tasks break down into subtasks whose progress rolls up into the parent. Every action on the board is written to the card’s activity log, so you can reconstruct its whole history after the fact: who moved it, when the progress changed, and why it ended up in a different sprint.

A day with aeman

Here’s what the whole flow looks like:

1. Morning sync. We open the Team board for yesterday, the team lead shares their screen, and we go through the cards: what’s done, what’s stuck, person by person, with the lead fixing statuses as we go. Then the lead hits Carry over and everything unfinished moves into today. Then we discuss what’s new: the lead creates cards on the fly and distributes them across people and zones.
2. The day. Everyone works from their own Me board. Something urgent lands — a card goes into the yellow zone. Something gets blocked — the Locked stage, and the lead sees it. Progress moves along the way; thoughts and findings go into the day’s notes.
3. Done — the card is at 100%, stage Done. If it needs review, Send to review, and it shows up for the reviewer.
4. Tomorrow it all repeats.

Every board is live: edits by colleagues and by AI agents appear on everyone’s screen in about a second, with no reload. More on how that works below.

Under the hood

Now for the part the engineering crowd came for.

GitHub Projects v2 as the only storage

aeman has no database of its own at all. Every card is an item on a GitHub Projects board, and every field — zone, progress, sprint, weekly plan — is an ordinary project field. aeman even provisions the fields it needs lazily: point it at any empty project and the first change creates whatever is missing. You could say aeman is a specialized view over a GitHub board: the same board opens in GitHub’s native interface, the data is always yours, and you can walk away from aeman at any moment without migrating anything.

A Kubernetes-style API

The aeman API is deliberately built on the same principles as the Kubernetes API — a pattern that has proven itself for synchronizing distributed state. Each entity — Card, Sprint, Ordering, Presence — is a separate resource with the familiar kind, metadata, and spec.

The client follows the familiar list + watch scheme. It first fetches a full snapshot of the board, then opens a WebSocket and starts receiving a stream of ADDED, MODIFIED, and DELETED events. After a full resync the server sends a special Sync frame that signals the local state is completely up to date.

In effect, every open browser tab works like a Kubernetes informer with its own local cache. Any change made by another user or by an AI agent shows up on every open screen in about a second, and your own changes aren’t echoed back to you. GET /api/v1 returns a machine-readable catalog of all available resources and endpoints, so clients and AI agents don’t need to know the shape of the API in advance.

The GitHub API is slow. What to do about it

The project’s main engineering problem: GitHub’s GraphQL mutations take hundreds of milliseconds, sometimes whole seconds; bursts run into secondary rate limits; and — surprise — GitHub’s read replicas lag behind writes by several seconds. Write straight through and the UI turns into a slide show: move a slider, wait; drag a card, wait.

So writes in aeman use a write-behind scheme, and the user never waits for GitHub. Every change is applied to the server cache instantly, becomes visible on all open boards at once, and only then goes to GitHub asynchronously. Writes run through a background queue with rate limiting and automatic retries on transient errors, to stay clear of GitHub API rate limits.

The queue can coalesce consecutive changes, much like the Kubernetes DeltaFIFO. If a user changes the same card field several times in a row, only the final value is sent to GitHub. Card content follows a different rule: text is never lost. If a user edits a description or adds notes in quick succession, the queue merges them into one final version and sends it in a single request. Until the write is confirmed, new notes exist under temporary identifiers, which are then swapped for real ones automatically.

If a write still fails after all the retries, the server notifies every connected client and rereads the board state from GitHub. GitHub remains the single source of truth. On shutdown, aeman first waits for the queue to drain, so that changes already confirmed to the user aren’t lost.

GitHub doesn’t guarantee that a change is available for reading the moment it’s written. Sometimes, right after a successful write, the API keeps returning the old state for a while. Because of that a user could see a freshly moved card jump back, a deleted card reappear, or a new one land in the wrong place. To avoid this, aeman treats its own recent writes as more trustworthy than fresh data from GitHub for a short window. Once GitHub starts returning the current state, the system switches back to it automatically. As a result, the user never sees the interface roll back.

Comments and the activity log

Notes and change history also live in GitHub, with no separate database. For draft cards the log sits right in the issue body, after a special <!-- aeman:log --> marker. If a card is linked to an existing issue or pull request, notes become ordinary GitHub comments and are visible to everyone in the discussion.

Every action on the board — creating a card, changing progress, moving between sprints, sending to review, and the rest — is recorded by the server as a machine-readable event line. To avoid burying subscribers under dozens of notifications, all those events are collected into a single log comment that gets appended to over time. The log is capped at the last 200 events, and user notes are never deleted.

The result is a complete change history for every card: who moved it and when, how the progress changed, and why it ended up in one sprint or another. That log is useful not only for audit, but also as a data source for statistics and metrics.

A pluggable backend

aeman was designed from the start for a pluggable backend. Today cards live in GitHub Projects, but the system itself isn’t tied to GitHub. All interaction with storage is hidden behind a Backend interface, so supporting a new backend comes down to implementing that one interface.

In future iterations I want to add a git backend that stores cards as plain files in the repository. Neither the user interface nor the application’s business logic will have to change.

The packages under pkg/ can also be used as an ordinary Go library, which lets you embed the board engine into your own tool. There’s more on that in docs/embedding.md.

MCP for AI agents

The same binary can also run as an MCP server. That makes Claude and other AI agents full participants in the process: they create and move cards, leave notes, run carry-over between sprints, and perform other operations.

There’s no separate API for AI and no special privileges. Every action goes through the same domain layer as a human action, with the same contract and the same checks, and the changes show up for everyone immediately.

Besides the local mode, aeman offers a publicly available MCP server. Engineers can connect AI clients without installing anything locally and start working with their tasks directly from Claude Code and other AI agents.

How to try it

The simplest way to try aeman is to run it locally with your own GitHub token.

gh auth login # the project and repo scopes are required
git clone https://github.com/aenix-io/aeman
cd aeman
make build
./aeman serve

The browser opens automatically at http://127.0.0.1:8765. You can use any GitHub Project as the board — preferably a new, empty one. aeman creates all the fields it needs on the first change.

For teamwork there’s a multi-user mode. The repository ships a ready-made docker-compose.yml, login through a GitHub OAuth App is supported, and every user works with their own GitHub token. Detailed instructions are in docs/deploy.md.

Where we landed

The whole company has been living on aeman for a few weeks now. Product managers and team leads quickly took to the new approach to planning, though — as with any new tool — there was some skepticism. But the main effect turned out to be a different one: in the morning every engineer knows what they’ll be working on today, unplanned work has stopped being invisible, and the daily syncs have become far more constructive and focused.

The project is open source under the Apache-2.0 licence and available on GitHub: github.com/aenix-io/aeman.

If you decide to try aeman, I’d be glad to hear any feedback, any problems you run into, and your stories about how planning works where you are.

Focus on today: how we built aeman, a daily planning board for engineers on top of GitHub Projects was originally published in Ænix on Medium, where people are continuing the conversation by highlighting and responding to this story.

Cozystack v1.6.0 was published on 22 July 2026. It replaces the Ubuntu and kubeadm bootstrap of tenant Kubernetes workers with Talos Linux driven by Cluster API, completes the etcd-operator v1alpha2 migration with in-place adoption of live clusters, adds OIDC single sign-on for tenant kube-apiservers and per-instance Grafana, introduces a tenant-facing SecurityGroup firewall API, and makes tenant resource quotas hierarchical. It rolls up every fix from v1.5.1 and v1.5.2.

It also carries the largest upgrade surface since v1.0. The platform migration targetVersion moves from 45 to 54, so migrations 45 through 53 run as pre-upgrade hooks — and three of them can block or wedge the upgrade if their preconditions are not met. Plan a window and read the checks below.

Read this before upgrading

Upgrade to v1.6.4 (or the latest 1.6.x), not v1.6.0. Three of the fixes that landed after v1.6.0 are the kind that fail silently:

• v1.6.0 shipped a fail-open copy of hack/seaweedfs-naming-audit.sh — the script whose output gates a runbook step that deletes PVCs. Every kubectl call in it was silenced with 2>/dev/null, so a timeout or an RBAC denial produced an empty “all clean” table byte-identical to an honestly clean fleet. Fixed in v1.6.1. Run the audit from a v1.6.1-or-later checkout, and read the exit code, not the table. A clean result from the v1.6.0 copy is not evidence of anything.
• v1.6.2 fixes Velero CRDs staying frozen at whatever version was first installed, because Helm never touches a chart’s crds/ directory on upgrade. Once the Velero image moved to a version with new backup phases, the apiserver rejected phase transitions against the stale CRDs and backups stopped while the HelmRelease stayed green.
• v1.6.2 also repairs the default backup Strategy CRs and the Velero BackupStorageLocation, which were gated on a Helm lookup performed while the referenced object was still being created. When the lookup came back empty the objects were skipped permanently, since helm-controller does not re-render a release whose chart and values are unchanged.

If you are coming from the 1.5 line, go through v1.5.4 first. v1.5.4 is the first 1.5.x release stamped targetVersion: 46, and its slot 45 stamps helm.sh/resource-policy: keep onto the CAPI KubeadmConfigTemplate objects. 1.6 drops KubeadmConfigTemplate from the tenant kubernetes chart entirely — workers move to TalosConfigTemplate — so without that pin Helm deletes the template while the kubeadm-backed MachineSet is still mid-rollover with its bootstrap.configRef pointing at it. Verify before upgrading:

kubectl get kubeadmconfigtemplates.bootstrap.cluster.x-k8s.io -A \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,KEEP:.metadata.annotations.helm\.sh/resource-policy'

A row showing <none> means the pin did not land. Fix it before going to 1.6.

Pre-upgrade checks

Run these against the management cluster before applying the v1.6 Platform Package.

1. etcd adoption needs a reachable backup target. Migration 50 adopts every legacy etcd.aenix.io/v1alpha1 cluster onto the new operator and takes a mandatory snapshot first. If it cannot resolve the snapshot target it exits 1 and halts the upgrade.

kubectl get etcdclusters.etcd.aenix.io -A
kubectl get etcds.strategy.backups.cozystack.io cozy-default-etcd
kubectl get secret cozy-backups-creds -n cozy-velero
kubectl get buckets.apps.cozystack.io cozy-backups -n tenant-root

If the first command prints nothing, migration 50 is a no-op.

2. SeaweedFS naming audit. v1.6.0 pins fullnameOverride: seaweedfs and adopts the running workload set in place, but two states cannot be adopted automatically and the chart fails the render rather than guess: class S (installed fresh on 1.5.x, data on data1-seaweedfs-system-volume-* PVCs) and class MIXED (both naming generations present). Both must be recovered before upgrading, following the seaweedfs-431-rename-recovery runbook. Class L needs nothing. A cluster that went 1.4.x straight to 1.6 never renamed and is unaffected.

3. Tenant clusters still on Kubernetes v1.30. v1.30 leaves the Talos-to-Kubernetes support matrix and the chart refuses to render it. Migration 46 patches live CRs to v1.31, but a GitOps-managed CR is overwritten again by the next source reconcile — so bump spec.version in Git.

kubectl get kuberneteses.apps.cozystack.io -A \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,VERSION:.spec.version'

4. Hand-made tenant StorageClasses that collide. Remote-accessible LINSTOR StorageClasses are now created inside each tenant cluster under the same name. A manually created tenant StorageClass with a colliding name — typically replicated — blocks the propagated class and stalls the tenant CSI release. Delete any such class that is not Helm-managed. Infra classes that must stay node-local need an explicit linstor.csi.linbit.com/allowRemoteVolumeAccess: "false"; an absent annotation is treated as remote-accessible.

5. Deprecated etcd module backup values. The backup.* block on the etcd tenant module is removed in this release. Move to a BackupClass bound to the Etcd strategy first.

6. Coming from v1.4.x? The v1.5.0 requirement still applies: Kubernetes 1.33 or newer on the management cluster and on any tenant cluster enabling the Flux addon.

Talos Linux for tenant Kubernetes workers

Tenant worker nodes no longer boot Ubuntu and bootstrap through kubeadm. Phase 1 of the Kubernetes-app split replaces that path with Talos Linux, driven by cluster-api-bootstrap-provider-talos (CABPT v0.6.12) registered as a second bootstrap provider alongside kubeadm, plus a clastix/talos-csr-signer sidecar embedded in the Kamaji control-plane pod.

Talos PKI — an Ed25519 CA plus trustd TLS — is generated through cert-manager with stable, lookup-and-reuse Talos secrets. The Kamaji control-plane provider carries an upstream-bound patch exposing KamajiControlPlane.spec.network.additionalServicePorts, so trustd (50001/TCP) can be published on the apiserver Service. Workers boot the Talos openstack raw image via a CDI DataVolume streamed from the image factory, with the system disk exposed as virtio-blk using blockSize.custom: logical=512, physical=4096 — so 4Ki-native backends such as LINSTOR/DRBD behave under QEMU’s O_DIRECT writes while SeaBIOS still boots.

The separate disk-kubelet PVC is gone. Talos lays out EPHEMERAL itself on the single system disk that nodeGroups[*].diskSize now sizes.

What this means operationally:

• Existing tenants roll over automatically. Old machines are replaced by Talos workers without manual intervention, and migration 45 pins the outgoing KubeadmConfigTemplate objects with helm.sh/resource-policy: keep so Helm cannot prune them mid-rollover. But plan for a full worker-pool replacement per tenant cluster: worker disks are reprovisioned and container images re-pulled.
• Worker MachineHealthCheck remediation is now ON by default. maxUnhealthy moved from a hard-coded 0 to nodeHealthCheck.maxUnhealthy, defaulting to "50%". CAPI now deletes and replaces unhealthy worker Machines. Set nodeHealthCheck.maxUnhealthy: "0%" to keep the old behaviour until your fleet is stable on Talos. Per-node-group maxUnhealthy and nodeStartupTimeout overrides also ship, so a stateful group can sit at 0% while a stateless one tolerates 50%.
• The default md0 node group is no longer merged into every cluster. nodeGroups defaults to {} and the built-in md0 applies only when no node groups are configured. Migration 47 pins md0 explicitly on existing CRs to preserve live topology; it is fail-closed, aborting the upgrade rather than letting Helm prune a live md0 MachineDeployment.
• Fresh clusters with nodeGroups: {} come up with zero workers. The chart no longer manages MachineDeployment.spec.replicas — the cluster-autoscaler owns it alone, seeded from minReplicas: 0. Supply a node group with roles: [ingress-nginx] and minReplicas >= 1, or let the autoscaler bring up md0 once ingress-nginx pods go Pending. The upside for existing clusters: helm upgrade no longer drains workers back to a hardcoded replicas: 2 on every platform bump.
• Air-gapped installations lose the registries.mirrors passthrough to tenant workers — the Helm-rendered *-patch-containerd Secret has no consumer in the Talos machine config and was removed. The Talos OS image and installer are overridable via talos.imageFactoryURL and talos.installerRepository, but in-guest registry mirroring is a Phase 2 follow-up.
• Worker node disks now default to the application-level replicated StorageClass. A node group leaving storageClass empty previously took the management-cluster default; it now falls back to the application storageClass, because linstor-csi v1.11.2 rejects ReadWriteMany volumes on a non-DRBD class and worker VMs need RWX to live-migrate. The rendered worker template changes, so each tenant’s worker MachineDeployment rolls once.

The tenant kubernetes HelmRelease now reports Ready as soon as Helm completes (DisableWait) and no longer blocks on worker or addon readiness; worker-rollout health is tracked by WorkloadMonitor instead.

etcd-operator v1alpha2 with in-place adoption

The etcd stack moves to the donated etcd-operator.cozystack.io/v1alpha2 API — a Membership-API lifecycle replacing the StatefulSet model — served by a Cozystack-authored chart at appVersion v0.5.2, with CRDs split into their own etcd-operator-crds package so they can be installed ahead of the controller.

The interesting part is the upgrade. Migration 50 runs as a pre-upgrade hook, while the legacy operator is still the running one, and uses the etcd-migrate tool baked into the migrations image to rewrite ownership, labels and CRs so the new operator takes over the live data plane on its first reconcile — no data move, no pod restart. Before touching anything it takes a mandatory snapshot of every adopted cluster to the cozy-backups bucket under a system key prefix with tenant-invisible credentials, and it fails loudly rather than adopt without one. It also re-issues each cluster’s server and peer certificates with the new operator’s native member wildcard SAN, because in secretRef TLS mode the operator never mints certs — an adopted cluster whose certs lack the native domain would silently fail TLS the first time a member is replaced, while still reporting Available=True.

If migration 50 halts on a cluster that genuinely has no backup storage, the documented escape hatch is:

kubectl edit package.cozystack.io cozystack.cozystack-platform

spec:
  components:
    platform:
      values:
        migrations:
          etcdAdoptSkipBackup: true

Flux re-renders the platform chart and re-runs the hook, adopting every legacy etcd without a snapshot. A botched adoption on a tenant control-plane etcd is unrecoverable without a snapshot. Set the flag back to false afterwards, or the next etcd migration skips its snapshot too.

SecurityGroup: tenant-managed network policy

A new tenant-facing, namespace-scoped firewall resource — sdn.cozystack.io/v1alpha1 SecurityGroup — lets tenants manage their applications’ network policy without any access to the cilium.io API group.

A SecurityGroup is a membership group. It owns a membership label (securitygroup.sdn.cozystack.io/<name>) that a new securitygroup-controller stamps onto the pods of every managed application listed in spec.attachments, and removes when an attachment is dropped. The group projects 1:1 onto a CiliumNetworkPolicy in the same namespace whose endpointSelector is exactly that membership label — so one SecurityGroup can cover several applications at once, and fromSG / toSG peers resolve group-to-group references live in the Cilium dataplane. A finalizer guards the backing policy so member labels are always stripped before it is removed, and the aggregated-API REST storage re-asserts them on every write so a full-replace PUT cannot orphan them.

The important limitation: ingress and egress rules only ever add allowed traffic. An empty rule list does not isolate the member pods, because effective connectivity remains the union of every policy selecting a pod — including the platform’s blanket-allow baseline. Default-deny enforcement is tracked separately as future work. Do not treat a SecurityGroup as a segmentation boundary in this release.

Hierarchical tenant resource quotas

tenant.spec.resourceQuotas previously became a plain per-namespace ResourceQuota that ignored the tenant tree entirely: a super-admin inside a quota’d tenant could create a sub-tenant with a larger quota, or with no quota at all, and consume more than was allocated. A tenant’s declared quota is now the budget for its whole sub-tree.

Enforcement is two cooperating layers, mirroring OpenShift’s ClusterResourceQuota split. A declaration-time gate in the aggregated apiserver deterministically rejects a sub-tenant whose declared quota exceeds the parent’s remaining budget. A runtime enforcer in cozystack-controller maintains a per-namespace tenant-quota-allocated ResourceQuota that clamps each pool member to its share, aggregating sub-tree usage so members collectively stay within it — Kubernetes enforces the most restrictive ResourceQuota in a namespace, so this binds without fighting Flux over the chart-rendered tenant-quota.

Review sub-tenant quotas before upgrading. A parent quota lowered below the sum of its children emits a QuotaOvercommitted event, and --tenant-quota-buffer-percent on cozystack-controller temporarily inflates pool budgets during rollout so workloads already over a freshly-binding quota keep running. There are no API type changes and no tenant chart changes — the feature works with the existing field.

OIDC single sign-on for tenant Kubernetes and Grafana

Two symmetric Phase 1 features let a tenant opt an individual workload into the platform’s Keycloak cozy realm through one flat selector, with no platform-level configuration.

Tenant kube-apiserver. Each Kubernetes CR gains spec.oidc.mode: System | CustomConfig | None, defaulting to None. System trusts the cozy realm via a per-cluster public client and audience binding; CustomConfig accepts a tenant-supplied structured AuthenticationConfiguration. spec.oidc.users[] drives one ClusterRoleBinding per user inside the tenant cluster (admin maps to cluster-admin, view to view). In System mode a <release>-oidc-kubeconfig Secret carrying a ready-to-use kubectl oidc-login exec block is surfaced through the Cozystack Dashboard, so a tenant user gets from “cluster exists” to “kubectl works with my SSO identity” without an operator in the loop. The underlying controlPlane.apiServer.extraArgs, extraVolumes and extraVolumeMounts passthrough on KamajiControlPlane is also exposed for anyone hand-rolling authentication.

Grafana. Each Monitoring CR gains the same selector. System wires a per-instance confidential Keycloak client with audience binding; authorization is app-side, with spec.oidc.users: [{email, role: Admin|Editor|Viewer}] reconciled into Grafana’s Main Org by a chart-owned post-install and post-upgrade Job that pre-provisions users, adds them to the org, patches their role and prunes stale members. CustomConfig accepts a tenant-supplied [auth.generic_oauth] payload, inline as a map or as a Secret with an auth.ini key — in the Secret case spec.oidc.users is unsupported and the chart fails the render on that combination rather than silently ignoring it. The admin_user / admin_password Secret remains a documented break-glass path in every mode.

Reference: enabling OIDC.

Application deletion now reclaims storage

Deleting a managed application used to leave its volumes and operator-generated Secrets behind. A ten-PR sweep across the catalog adds post-delete cleanup hooks and PVC retention policies so a deleted application actually releases what it held: ClickHouse keeper, data and log PVCs; Qdrant’s data PVC; OpenBAO’s data PVC; the tenant monitoring module’s VictoriaMetrics and VictoriaLogs storage and TLS Secrets; the tenant seaweedfs module’s volume PVCs; the tenant etcd module’s data-etcd-* PVCs; Harbor’s jobservice and trivy PVCs; MariaDB’s operator-generated password Secrets; Bucket and Gateway ACME resources.

This makes deletion irreversible. That is the point of the change, but it is a real behaviour change from “delete leaks the volume, and you could recover from it”. Harbor’s PVCs are force-deleted, overriding persistence.resourcePolicy=keep; Harbor’s CNPG database PVCs are not touched. Migrations 48 and 51 backfill the release label onto pre-existing ClickHouse keeper and monitoring storage PVCs so already-deployed clusters are covered; both are best-effort, and the worst case if a relabel is skipped is the pre-existing leak.

Back up before deleting. Tell your tenants.

Also in v1.6.0

• Wildcard certificates, end to end. v1.5’s publishing.certificates.wildcardSecretName covered the root tenant only. The platform controller now replicates that certificate into every tenant namespace that terminates TLS, so per-tenant ingress controllers and Gateways serve it automatically, with no cross-namespace Secret read. A new opt-in publishing.certificates.wildcard (default false) has the platform mint a single *.<root-host> certificate via DNS-01 on the default ingress-nginx path — the practical fix for hitting Let’s Encrypt rate limits at scale.
• Keycloak gains four independent opt-ins: a KMS-encrypting database proxy for column-level PII encryption at rest, backed by a static KEK or Vault Transit (with Vault Kubernetes and AppRole auth); an ingress.adminHost that serves the admin console and Administration REST API on a separate hostname and route, attachable to a private Gateway or ingressClass via publishing.ingressNameAdmin; Barman-based S3 backups of the CNPG database; and a selectable realm login theme shipped through platform branding values. All are off by default.
• Immutable tags and rc-to-stable promotion. A stable release is now the byte-identical promotion of the release candidate that was tested: no tag is ever force-moved, stable is never rebuilt, and images are retagged by digest. The cron-driven auto-release.yaml patch-tag workflow is deleted outright.
• Kamaji defaults to 2 replicas with soft pod anti-affinity, and its telemetry handler and webhook are removed — cutting apiserver admission p99 for TenantControlPlane webhooks by roughly 70% on multi-tenant clusters.
• Remote-accessible LINSTOR StorageClasses propagate into tenant clusters under the same name, with the class named by storageClass (default replicated) becoming the tenant default and the legacy kubevirt class kept as an alias.
• The Cozystack Dashboard now shows the LoadBalancer external IP on the application Services tab, and renders explicit error and unknown-type states instead of an infinite spinner. The console SPA is now built from in-tree source; the shipped image and digest are unchanged.
• After upgrading, re-check metrics scrape targets on port 10250 — cert-manager’s webhook, external-secrets’ webhook and metrics-server’s listener all moved off it.

Platform components

Talos system extensions are refreshed to the 20260622 set, including DRBD 9.3.2 and ZFS 2.4.3. Applying that alone rolls each tenant worker pool once, since the worker template is renamed to carry the new image. Managed Kubernetes patch versions move to v1.32.13, v1.33.13, v1.34.9 and v1.35.6; v1.30 is out of the tenant support matrix.

Two API groups are worth knowing about as non-events. SecurityGroup had spec.targetRef replaced by the spec.attachments[] membership model before release, so only people tracking main between 30 June and 16 July 2026 need to rewrite objects. The network.cozystack.io group (ExposureClass / ServiceExposure) was introduced and removed within the cycle and is not part of v1.6.0 — native Service type: LoadBalancer plus loadBalancerClass covers the same ground, exposed as publishing.loadBalancerClass.

The 1.6 patch line

• v1.6.1 (5 August 2026): CNPG operator and CRDs aligned to 1.28.2, fixing a PVC resize deadlock that could leave a single-instance PostgreSQL cluster with zero instances; the talos-reconcile Job now renders for the implicit md0 group, so the first autoscaler-driven scale-up no longer leaves Machines permanently blocked with no matching TalosConfigTemplate; the keycloak-configure pre-delete Job patches the HelmRelease in the right namespace, unblocking Keycloak uninstall and reinstall; the SeaweedFS naming audit fails closed; etcd-operator to v0.5.4.
• v1.6.2 (19 August 2026): the backup-strategy lookup gate, the Velero CRD upgrade, and the kube-ovn webhook certificate reload — that last one mattering because kube-ovn-webhook loaded its serving certificate once at startup and never re-read it, so after a cert-manager renewal every pod creation in tenant namespaces was rejected under failurePolicy: Fail, including virt-launcher pods, blocking VMI startup. Also: barman-cloud requests an S3 checksum only when required, so backups to Ceph RGW and some MinIO and R2 builds stop failing outright; sharded helm-controllers no longer crashloop behind an HTTP proxy; kubectl apply --validate works again against the core and sdn API groups.
• Later patch releases v1.6.3 and v1.6.4 followed. Use the latest 1.6.x; see the Cozystack releases.

Upgrading

kubectl annotate namespace cozy-system helm.sh/resource-policy=keep --overwrite
kubectl annotate configmap -n cozy-system cozystack-version helm.sh/resource-policy=keep --overwrite

helm upgrade cozystack oci://ghcr.io/cozystack/cozystack/cozy-installer \
  --version 1.6.4 \
  --namespace cozy-system

The annotations are required — without them, removing or upgrading the installer release can delete the cozy-system namespace and everything in it. Reuse your existing values, watch the migration hooks, and expect the rollouts listed above: tenant worker pools, the dashboard and linstor-gui gatekeepers, the VPA admission controller, and the linstor-scheduler admission Deployment. Full procedure: upgrade guide.

Where Ænix fits

Cozystack is a CNCF Sandbox project under Apache 2.0. v1.6 is a good release and a demanding upgrade — the Talos worker rollover, the etcd adoption and the deletion-semantics change all deserve a rehearsal on a non-production cluster first. Ænix created Cozystack and is one of its maintainers; it sells enterprise support for Cozystack, including upgrade planning and hands-on migration for teams running it at scale.

Release links

• Cozystack v1.6.0 on GitHub
• Cozystack v1.6.2 on GitHub
• All Cozystack releases on GitHub
• Cozystack v1.6 documentation
• Telegram and Slack (invite at slack.kubernetes.io)

Cozystack v1.5.0 was published on 22 June 2026. It rolls up every fix from the v1.4.1 to v1.4.4 patch line and pushes the platform in five directions: a second ingress path via Gateway API, backups that work without per-app S3 configuration, stricter and shardable Flux reconciliation, TLS on externally published managed services, and GPU passthrough that no longer needs a manual kubectl patch.

This release also has a real upgrade surface. Read the next section before you plan the window.

Read this before upgrading

Do not install v1.5.0, v1.5.1 or v1.5.2 today. Go to v1.5.4. The v1.5.0 SeaweedFS chart bump (4.05 to 4.31) renamed SeaweedFS workloads from chart-based names to release-based names. StatefulSet names are immutable, so Helm could not rename in place — it stood up a second, duplicate set beside the running one. On a cluster with more nodes than master replicas the empty duplicate schedules successfully, both generations carry identical pod labels, and the seaweedfs-s3 Service load-balances across two filers writing into the same metadata store. That is a data-integrity incident, not a cosmetic duplicate. A related defect in migration 43 could prune the seaweedfs-db CNPG cluster — the index for every object in a tenant’s S3 — for any instance not literally named seaweedfs.

Both are fixed in v1.5.4 (19 August 2026), which pins fullnameOverride: seaweedfs, adopts the running workloads in place, and ships a fail-closed audit script. v1.5.4 is the final release of the 1.5 line and the only one worth deploying.

Five further changes need attention on any 1.5 upgrade:

• Kubernetes 1.33 or newer is required on the management cluster, and on any tenant cluster that enables the Flux addon. Flux v2.8’s helm-controller v1.5 requires it. Upgrade Kubernetes first.
• upgrade.force: true is gone. Immutable-field changes no longer self-heal. If a chart upgrade changes a StatefulSet volumeClaimTemplates or serviceName, the apply fails; recreate the object by hand with kubectl delete sts <name> --cascade=orphan and let Flux reconcile.
• GPU VM operators must move custom host devices first. With cozystack.gpu-operator enabled, the bundle now owns KubeVirt.spec.configuration.permittedHostDevices and overwrites it on the first reconcile. Move hand-edited entries into .gpu.permittedHostDevices before upgrading and confirm every resourceName matches what the nodes advertise.
• MetalLB moves to the FRR-K8s BGP backend and HTTPS-only metrics. MetalLB v0.15.2 to v0.16.1 adopts the upstream-default FRR-K8s backend; classic FRR mode is deprecated. kube-rbac-proxy is replaced by native TLS plus RBAC, so any scrape config pointing at the old plain-HTTP metrics endpoints must be updated. The host-network port denylist is rotated to match, including the new /healthz and /readyz port 17472.
• Externally published databases and messaging gain TLS automatically. Instances with external: true flip to TLS-on after upgrade. The trust anchor is a self-signed CA, so external clients must retrieve and pin it. Cluster-internal instances are unaffected.

One more, if you plan to go on to 1.6 later: v1.5.4 is the first 1.5.x release stamped migration targetVersion: 46, and the pin it applies to the CAPI KubeadmConfigTemplate objects is what keeps a later 1.6 upgrade from pruning them out from under a live, mid-rollover MachineSet. Reaching 1.6 from anything older on the 1.5 line is the path that hurts.

Gateway API via Cilium

Cozystack-native services can now be published through the Gateway API backed by Cilium, as an opt-in alternative to the per-tenant ingress-nginx controllers. It is materialised per tenant by a new gateway.cozystack.io/v1alpha1 TenantGateway CRD reconciled by cozystack-controller.

Enable it platform-wide with publishing.gateway.enabled=true. A tenant then either gets its own Gateway, LoadBalancer IP and certificate with tenant.spec.gateway=true, or inherits the nearest ancestor’s Gateway through the same label-based selector model that already drives ingress inheritance. Two certificate solver modes ship: HTTP-01 (the default — a per-app certificate, no platform configuration for new apps) and DNS-01 (opt-in — one wildcard certificate covering an apex, with Cloudflare, Route 53, DigitalOcean and RFC 2136 providers).

Defaults stay on ingress-nginx, so existing clusters do not change behaviour. Two side effects do land on everyone:

• Cilium Envoy and Gateway API support are now always enabled. That is an extra cilium-envoy DaemonSet, roughly 100 MB RAM per node at idle. Budget for it on dense nodes.
• cozystack-api now invokes admission (createValidation / deleteValidation) on Create and Delete for apps.cozystack.io/*. Any custom ValidatingAdmissionPolicy or webhook on those kinds now fires on all three verbs.

Reference: Gateway API guide.

Backups that work out of the box

Previous releases installed the backup machinery. v1.5 closes the gap between “installed” and “working without per-app S3 configuration”.

A platform-managed default BackupClass named cozy-default now ships, backed by a system bucket cozy-backups. Applications opt in with useSystemBucket, after which the platform projects shared backup credentials into the tenant namespace with RBAC isolation and projection metrics, and skips per-release credential Secrets entirely. Default strategies are provided for every backup-capable app — Velero for VMDisk and VMInstance, CNPG for PostgreSQL, Altinity for ClickHouse, plus MariaDB, FoundationDB and etcd — and a Velero BackupStorageLocation is wired to the system bucket. The legacy per-tenant S3 fields on Postgres and ClickHouse are deprecated in favour of this flow.

Velero is now a default system package, not an optional one. This fixes a deterministic failure: the default backupstrategy-controller hard-depends on Velero, so on clusters without it the controller sat in DependenciesNotReady and kept the platform HelmRelease from ever reaching Ready. Existing clusters get Velero in the cozy-velero namespace on upgrade. If you do not back up VMs, opt out via bundles.disabledPackages.

Two new strategies join the catalog. An etcd strategy (cluster-scoped strategy.backups.cozystack.io Etcd, S3-only) with a snapshot BackupJob and a destructive in-place RestoreJob. And a generic, application-agnostic Job strategy: the operator supplies a Kubernetes Job template, Cozystack renders and runs it as a one-shot backup, then re-renders it with .Mode == "restore" for recovery. Cross-namespace restore is not supported by the Job strategy.

References: application backup and recovery, managed app backup configuration.

Flux v2.8 and helm-controller sharding

Flux moves from v2.7.3 to v2.8.0 across both the embedded management-cluster Flux and the optional tenant Flux addon; the flux-operator and flux-instance charts go v0.33.0 to v0.50.0.

helm-controller v1.5 ships Server-Side Apply with --force-conflicts and kstatus-based health checking by default. Two practical consequences: chart fields that v2.7 silently dropped are now hard errors (fixed in this release for foundationdb, kafka, kubevirt-instancetypes, vm-instance and the platform chart), and parent HelmReleases now wait for every child resource to be Ready before reporting Ready themselves. Correctness improves; first-install timings get longer and stricter, which is why several packages gained explicit spec.timeout and dependsOn entries in the same release.

Alongside it, a new flux-shard-operator spreads tenant HelmReleases across multiple helm-controller shards, so one noisy tenant — the canonical case being a HelmRelease stuck in infinite remediation — can no longer degrade reconciliation for everyone else. Placement is per tenant: all of a tenant’s HelmReleases share one shard, assigned greedily by least load, with a CREATE-time mutating webhook stamping the shard label. It defaults to shardCount: auto, which sizes shards from the tenant HelmRelease count — small clusters stay on a single shard, large fleets shard out automatically — and an integer pins the count. The hand-rolled flux-tenants deployment is drained and retired by migration 44.

TLS for managed databases and messaging

Four managed-app charts gain TLS through a single tls.enabled value with consistent tri-state semantics. Unset, it inherits external: TLS is on when the service is published externally and off when it is cluster-internal. An explicit true or false always wins. In every case the trust anchor is a chart- or operator-managed self-signed CA that clients retrieve and pin — there is no publicly trusted CA in this path.

• Kafka serves TLS on its external LoadBalancer listener (port 9094), certificates managed end to end by the Strimzi operator. Clients trust via the operator-published <release>-cluster-ca-cert and <release>-clients-ca-cert Secrets. The external listener is now gated only on external: true, decoupled from tls.enabled.
• NATS and Qdrant use a self-contained cert-manager chain (self-signed Issuer, CA, leaf) rendered in the tenant namespace. NATS covers client connections and cluster routes; Qdrant covers REST and gRPC. Clients trust the <release>-ca Secret.
• PostgreSQL already served TLS unconditionally via CNPG, so tls.enabled here injects the external hostname into the operator-managed server certificate’s SANs when external: true — which is what makes sslmode=verify-full work against the external endpoint. Clients read ca.crt from the <release>-credentials Secret.

The TLS work in this release stops at these four charts. The other managed services — MariaDB, Valkey, ClickHouse, OpenSearch, RabbitMQ, MongoDB — are unchanged.

GPU passthrough without manual patching

GPU enablement is wired across all three paths a GPU can reach a workload, each of which previously needed manual reconciliation.

For tenant Kubernetes, node groups declaring gpus automatically get the gpu=on kubelet label, so HAMi’s device plugin schedules and advertises nvidia.com/gpu. The tenant GPU operator loads the driver with NVreg_NvLinkDisable=1, which fixes single-SXM-GPU passthrough that previously hung at “Fabric State: In Progress” with CUDA reporting “system not yet initialized”. Both defaults are overridable via addons.gpuOperator.valuesOverride.

For KubeVirt VMs, enabling cozystack.gpu-operator auto-populates the KubeVirt CR: it injects the HostDevices feature gate and fills permittedHostDevices (plus mediatedDevicesConfiguration for vGPU) from shipped NVIDIA default tables. GPU VMs now schedule without a manual patch — at the cost of the ownership change listed in the upgrade notes above.

A third container variant of the gpu-operator is added for hosts where the NVIDIA driver and container toolkit are already installed by the OS; it exposes GPUs to ordinary containerized pods through the device plugin only.

GPU sharing in Cozystack remains NVIDIA GPU Operator plus HAMi. Reference: GPU sharing and operator variants.

Also in v1.5.0

• Deletion protection. Objects labelled platform.cozystack.io/no-delete=true cannot be deleted. The check runs in-process in the kube-apiserver through a ValidatingAdmissionPolicy — no webhook, no DaemonSet, no TLS, no extra image. Protected in this release: the cozy-system and tenant-root namespaces, the tenant-root HelmRelease, the cozystack-version ConfigMap, the cozystack-packages OCIRepository, the cert-manager ClusterIssuers, the LinstorCluster, and the packages CRDs. To delete one, remove the label first: kubectl label <kind> <name> platform.cozystack.io/no-delete-. Requires Kubernetes 1.30+.
• Runtime-populated dashboard dropdowns. A namespaced, read-only Option resource (core.cozystack.io/v1alpha1), computed on read by a privileged in-process provider registry, plus an x-cozystack-options schema keyword that charts declare. GPU devices, KubeVirt instancetypes and preferences, Multus networks, VM images, storage pools, storage classes, backup classes and plans become real dropdowns in the Cozystack Dashboard instead of free text or stale static enums. Tenants get read-only access to options in their own namespace.
• Tenants can start, stop and restart their own VMs. The missing update on the virtualmachines/start, /stop and /restart KubeVirt subresources is granted at cozy:tenant:use:base; the dashboard power buttons previously returned 403 for every tenant role.
• Tenant Overview Grafana dashboard for platform admins, deployed only to the root Grafana in cozy-monitoring and never to per-tenant Grafanas, plus a Cluster Usage admin page backed by a dedicated cozystack-dashboard-cluster-usage ClusterRole (cluster-wide and per-node utilization, GPUs included). The sidebar entry is fail-closed without the binding.
• storageClass marked immutable on 16 stateful apps — changing it never migrates data, because PVCs pin storageClassName at creation. Enforcement in v1.5.0 is UI-only: the aggregated apiserver does not yet evaluate the CEL rule on Update, so a direct kubectl patch is still accepted.
• Notable fixes: cozystack-api now publishes free-form .spec as x-kubernetes-preserve-unknown-fields rather than additionalProperties: true, which was crashing kube-controller-manager cluster-wide with a nil-pointer panic in the VAP type-checker; tenant kubeconfigs use the root host for the Keycloak OIDC issuer, so kubectl oidc-login no longer fails TLS verification for non-root tenants; config_path registry configuration works on containerd 2.x tenant nodes; RWX Block volumes are routed to the upstream hotplug detach path instead of the NFS-cleanup branch; OpenSearch is finally referenced by the PaaS bundle.

Platform components

kuberture is off by default. It bridges an external-dns gap — external-dns cannot read EndpointSlices — by watching the default/kubernetes API-server EndpointSlice and emitting annotated headless Services that external-dns consumes to publish the Kubernetes API endpoint to DNS. Enable via bundles.enabledPackages with at least one config.outputs entry.

The 1.5 patch line

• v1.5.1 (24 June 2026) fixes a v1.5.0 regression: persistent EFI/TPM state was restored for the windows.11, windows.2k22 and windows.2k25 KubeVirt preferences, which made KubeVirt provision a ReadWriteOnce persistent-state-for-<vm> PVC on the replicated StorageClass. That pins the VM to its node and blocks live migration and node drains — on clusters using evictionStrategy: LiveMigrate it can stall a cluster upgrade outright. Secure Boot and the vTPM still work; only the state persistence across reboots is dropped.
• v1.5.2 (3 July 2026) covers ten fixes, including a Kamaji DataStore deletion deadlock that wedged tenant namespaces, a victoria-metrics-operator dependency that could never resolve when certManager.enabled: false, and single-replica MariaDB instances being rejected by the operator webhook.
• v1.5.3 was tagged but its GitHub release was left a draft and never published. No operator received it.
• v1.5.4 (19 August 2026) is the final 1.5 release and the one to deploy.

Upgrading

kubectl annotate namespace cozy-system helm.sh/resource-policy=keep --overwrite
kubectl annotate configmap -n cozy-system cozystack-version helm.sh/resource-policy=keep --overwrite

helm upgrade cozystack oci://ghcr.io/cozystack/cozystack/cozy-installer \
  --version 1.5.4 \
  --namespace cozy-system

The annotations are required, not optional — without them, removing or upgrading the installer release can delete the cozy-system namespace and everything in it. Reuse your existing values. Full procedure and post-upgrade checks: upgrade guide.

Where Ænix fits

Cozystack is a CNCF Sandbox project under Apache 2.0, and the upgrade paths above are the same for everyone running it. Ænix created Cozystack and is one of its maintainers; it sells enterprise support for Cozystack — including upgrade planning for the sharp edges in this release — for teams that would rather not carry the platform alone.

Release links

• Cozystack v1.5.0 on GitHub
• Cozystack v1.5.4 on GitHub
• Cozystack v1.5 documentation
• Telegram and Slack (invite at slack.kubernetes.io)

Why white-label cloud matters for MSPs

MSPs have customer relationships hyperscalers can’t easily replicate. They lack the cloud product to monetize those relationships at scale. White-label cloud — branded as MSP’s product, running on shared or dedicated infrastructure — bridges this.

Pattern in 2026: MSP gets branded multi-tenant cloud product on open-source platform; customers consume MSP-branded cloud; MSP collects margin between platform cost and customer pricing.

Architecture

• Multi-tier Tenant CRD — root tenant → MSP tenant → MSP customer tenant. Per-tier isolation.
• Branded Cozystack Dashboard — MSP can customize colors, logo, domain, service catalog options (white-labelling is part of open-source Cozystack; Ænix support covers it from the Standard tier)
• WHMCS integration — billing flows through MSP’s existing customer-management system via the Ænix WHMCS integration, a proprietary Ænix module
• Service catalog — MSP can curate which services to expose to customers (e.g., hide Kafka if MSP doesn’t support it)
• SLA management — per-customer SLA tracking through Cozystack observability

Reseller economics

Typical economics for an MSP running white-label cloud:

• Platform cost — Ænix engagement + hardware + colocation
• Per-customer cost — incremental hardware/storage/bandwidth
• Customer pricing — typically 30-50% above raw platform cost
• Margin — covers MSP support, sales, operations

Break even at 30-50 paying customers when you are covering the platform and tooling; 50-100 when a dedicated on-call rota is funded alongside it. Positive economics after that, depending on customer mix.

Engagement structure

• Free 30-minute discovery call, then a fixed-price Platform Readiness Assessment (14 days focused or 28 days full)
• Platform live in weeks once hardware is ready, using the productized installer; branding, catalogue curation and billing integration follow at your pace
• Support subscription — published tiers per 10 nodes per month; for a white-label product, plan on Standard ($3,000) or higher (see /pricing/)
• Optional managed services

The honest test

Cozystack fits when at least three of:

1. Regulated data — banking, healthcare, public sector data with sovereignty / residency requirements
2. Multi-tenant model — SaaS / customer-facing / cross-BU isolation needed
3. Sustained workloads — 24/7 utilization where dedicated economics beat hyperscaler
4. Internal platform team — capacity to operate (or building it)
5. AI/GPU workloads at scale — sustained inference / training
6. Specific exit trigger — VMware migration, hyperscaler repatriation

If you have 0-1 of these, Cozystack is over-engineering. If 2, marginal. If 3+, fits.

When it doesn’t fit — what to do instead

For SMB without regulated data

• Hyperscaler-managed deployments (AWS, Azure, GCP) — operationally simple
• DigitalOcean / Hetzner / OVHcloud — managed cloud-adjacent
• Proxmox VE for on-prem virtualization

For mid-market with simple needs

• Vanilla Kubernetes if container-only — lighter than Cozystack
• Existing managed cloud if it works — don’t fix what isn’t broken
• Hetzner cloud + VPS if team is small — operationally simple

When it does fit — examples

Mid-market with regulated data

• Regional bank with multi-jurisdictional operations under DORA
• Mid-size insurance carrier with claims-processing AI on regulated data
• Public-sector or quasi-public with procurement-mandated sovereignty

Mid-market becoming multi-tenant

• SaaS company with 100+ customers needing hard isolation
• B2B platform serving regulated industry customers
• Specialty cloud product for vertical market

Mid-market with strong platform team

• Mid-market with intentional in-house platform engineering investment
• Fast-growing tech-mid-market scaling beyond hyperscaler simple model

Ænix engagement model for mid-market

• 30-minute discovery call — free, no sales pressure
• Platform Readiness Assessment (fixed price, 14 days focused) — if you want a structured assessment
• Implementation — only if it actually fits; self-run Cozystack with Ænix enterprise support is often enough at mid-market scale

For most SMB outreach, the honest answer is “stay where you are.” We’re explicit about this.

After Broadcom, the VMware bill stopped being predictable. Subscription-only licensing, mandatory VCF bundling, two-to-five-times price increases on renewal, and the end of perpetual licences changed the math for every infrastructure team running VMware at scale. The result has been a documented wave of VMware replacement projects across service providers, banks, government, telecom, and AI/GPU operators evaluating how to exit VMware safely.

VMware competitors at a glance

If you’re scoping the market, here is how Cozystack compares to the most-cited alternatives to VMware:

• Cozystack (this page) — open-source, Kubernetes-native, multi-tenant. VMs + containers + databases + object storage + GPU on one platform. Best for service providers and regulated enterprises that need sovereignty and want one stack.
• Nutanix AHV — proprietary KVM-based hypervisor inside Nutanix HCI. Strong VM-only story, weak on Kubernetes-native multi-tenancy, expensive at scale.
• Proxmox VE — open-source KVM + LXC. Excellent for SMB and labs; thin on managed databases, multi-tenancy, and enterprise support model.
• Scale Computing HC3 — appliance-based hyperconverged stack. Good for ROBO/edge; closed ecosystem.
• Red Hat OpenShift Virtualization — KubeVirt-based, similar foundations to Cozystack. Heavier OpenShift footprint and Red Hat licensing model.
• OpenStack — broad, mature, large operational surface area. Best when you have a dedicated team running it.
• Microsoft Azure Stack HCI — Hyper-V on validated hardware. Locks into Microsoft licensing.
• Verge.io / Spectro Cloud / Platform9 — vendor-led KubeVirt or hyperconverged stacks; comparable to Cozystack on architecture, different commercial models.

The rest of this page goes deep on Cozystack as a Cozystack-specific VMware replacement. For a head-to-head comparison of VMware alternatives, see VMware alternatives.

Why teams replace VMware in 2026

The technical case for moving off VMware existed before Broadcom. Broadcom turned it into a board-level decision.

1. Unpredictable, subscription-only economics

VCF subscription pricing replaced perpetual licensing. Renewal quotes have come back at 2× to 5× prior spend across our pipeline. ELAs have been broken or restructured mid-term. Standalone vSphere SKUs were retired in favour of bundled VCF tiers that include components most customers do not need.

For service providers, this collapses margin: end-customer prices are sticky, licence costs are not. For banks and regulated enterprises, it breaks multi-year capex planning that was built around perpetual entitlements.

2. Vendor lock-in across the stack

VMware’s strength was integration: vSphere, vSAN, NSX, vRealize, Horizon, vCD, all assumed each other. That same integration is now the lock-in surface. Replacing one component meant rebuilding adjacent ones, so most teams stayed.

Cozystack inverts this. Each layer is an independent open-source project (KubeVirt, LINSTOR/DRBD, Cilium, Kube-OVN, KubeVirt CDI, SeaweedFS, Velero, Flux, etc.), composed by a Kubernetes operator. You can replace any layer without rewriting the rest, and you can audit every line.

3. Sovereignty, regulator pressure, and US-vendor risk

DORA (in force across the EU since January 2025) and NIS2 require demonstrable control over critical ICT third parties. For European banks, telcos, and government workloads, depending on a US-headquartered closed-source hypervisor stack is a documented operational risk.

Cozystack is open source under Apache 2.0. Your binaries, your hardware, your data plane. Ænix ships air-gap install workflows and an advisory support model that does not require direct customer-environment access.

4. Pricing isn’t the only cliff — capability is

VMware’s roadmap is now Broadcom’s roadmap. KubeVirt and the Kubernetes-native virtualization stack have a community of hundreds of contributors and a release cadence VMware can no longer match in the open. GPU support, multi-tenant tenancy, modern storage replication, GitOps-native ops — all moved to Kubernetes-native projects years ago. VMware tries to retro-fit them through Tanzu and VCF.

What Cozystack gives you instead

Cozystack is a single platform you install on bare metal. Once it’s up, you have:

• Virtual machines through KubeVirt — full KVM-based VMs with live migration (CPU only), block-storage attach, snapshots, and templates.
• Tenant Kubernetes clusters for customers who want containers — every tenant gets their own K8s, isolated.
• Managed databases — PostgreSQL, MariaDB, MongoDB, Redis, Valkey, RabbitMQ, Kafka, NATS, ClickHouse, OpenSearch, Qdrant, FoundationDB — exposed as cloud services your tenants self-provision.
• S3-compatible object storage — for backups, application data, and AI training sets.
• GPU as a service — for VMs (passthrough of whole GPUs, or NVIDIA vGPU, which requires your NVIDIA vGPU licence) and for Kubernetes pods (whole-GPU through the NVIDIA GPU Operator, fractional sharing through HAMi). NVIDIA data-centre GPUs are supported through the NVIDIA GPU Operator; MIG and time-slicing are on the roadmap.
• Multi-tenant control plane — Tenant Kubernetes CRD, nested tenants, per-tenant quotas and presets.
• Observability built in — VictoriaMetrics + VictoriaLogs, open source and included.
• Backup and DR — Velero + S3 + per-database point-in-time recovery for managed services.
• Self-service portal — Cozystack Dashboard. Billing runs in your own system or through the Ænix WHMCS integration, a proprietary Ænix module (two integration modes) that is not part of open-source Cozystack.

It runs on your bare metal. No public cloud dependency. No phone-home telemetry by default.

VMware-to-Cozystack architecture mapping (vSphere alternative, ESXi alternative, vCenter alternative — all in one)

The questions every VMware admin asks first: “What replaces vSphere? What’s a real ESXi alternative? What replaces vCenter, NSX, vSAN, vCloud Director?” Here is the direct one-to-one mapping.

Two areas need redesign rather than one-for-one mapping: networking (Cilium is fundamentally different from NSX) and multi-tenancy model (Cozystack tenants are Kubernetes-native; vCD orgs are vSphere-native). For both, we run an architecture review before any migration commits.

How VMware migration actually works

Migration paths are workload-dependent. For most teams, this is the realistic sequence.

1. Discovery and assessment

We run a structured assessment of the current vSphere / VCF / vCD inventory: workload count, OS mix, vSAN / NSX dependencies, integrations (backup, identity, monitoring), and tenancy model. Output is a migration plan with workload buckets, risk flags, and timing.

2. Cozystack deployed in parallel

Cozystack is installed on new or repurposed hardware alongside the existing VMware estate. No big-bang cutover. Tenants migrate cohort by cohort.

3. VM-by-VM image migration to KubeVirt

For most VMs, migration is a disk-image copy. We use KubeVirt CDI plus a set of dedicated migration scripts we’ve built and reused across customer deployments. For Windows VMs and VMs with VMware Tools dependencies, we run an automated cleanup pass before boot on KubeVirt.

4. Network and storage cutover

Networking: VLAN mapping into Cilium, with policy parity checked against the source NSX rules. Storage: import disks into LINSTOR, validate IOPS and replication.

5. Validation and DR cutover

Each migrated workload runs in parallel on Cozystack until validated by the application owner. Backup-and-restore runbooks (Velero, PostgreSQL PITR) are written and rehearsed before final cutover; there is no SRM-style orchestrated cross-site failover.

6. VMware decommission

Licences end on their own terms. Hardware repurposed into the Cozystack cluster (we run on commodity x86 — your existing servers usually qualify).

For OpenStack, CloudStack, and Proxmox sources the same playbook applies, with different image-import and network-mapping steps.

Looking for a checklist? Use the VMware migration checklist. A structured assessment is part of the engagement — book a call to scope it.

Multi-tenancy and sovereignty

Cozystack was built for service providers first. The same model works for any organization that needs hard separation between business units.

• Tenant CRD — each tenant is a Kubernetes object. Namespaces are derived (<parent>-<name>), nested tenants are supported, quotas are enforced via Kubernetes ResourceQuotas.
• RBAC by default — tenants cannot see each other. Tenant operators get a scoped Kubernetes API.
• Air-gap install — supported and documented; works behind Harbor / Nexus / proxy patterns.
• No phone-home — telemetry is opt-in and disabled by default.
• DORA / NIS2-aligned controls — operational resilience, incident reporting, supplier-risk documentation. Architecture maps to the controls; certification, where required, is the customer’s audit process to run.
• Ænix support model — access is your choice. Advisory, runbooks and GitOps PR review need no access to your production cluster; remote access to your clusters, with your approval, and external monitoring are available on the higher support tiers. Critical for banks, telcos, and any regulated buyer who cannot expose infrastructure to a vendor.

Cozystack vs VMware — feature comparison

FAQ

Can we keep our existing hardware?

Yes — in most cases. Cozystack runs on commodity x86. The standard scenario is to deploy Cozystack on a new pod of servers, migrate workloads off VMware, then repurpose the freed VMware hardware into Cozystack as licences lapse.

How long is a typical migration?

Two numbers matter, and conflating them is how migration plans go wrong. The first production cohort typically runs in a live environment 6-12 weeks after kickoff — that is when the platform stops being a proof of concept. The whole estate takes about 8-12 months for ~100 VMs and 18-24 months for ~1,000 VMs, including planning and migration waves, after a 14- or 28-day Platform Readiness Assessment; mid-size estates fall in between, depending on dependencies. Small, flat estates land at the short end; vCD- or NSX-heavy regulated ones at the long end, run in cohorts. The driver is rarely raw migration speed — it’s regression testing and the parallel-run windows application owners will agree to.

Do you support Windows VMs?

Yes. KubeVirt runs Windows. We have an automated cleanup step that removes VMware Tools and installs the right virtio drivers before first boot on KubeVirt. Activation handling depends on your Windows licensing model — discussed during assessment.

What happens to our existing VMware operational skills?

The hypervisor concepts (VMs, snapshots, templates, networking) carry over. The control-plane shifts from vCenter to Kubernetes — for most teams this is a 4–8 week ramp with the right training and runbooks. Ænix runs the training as part of professional services.

What about VMware Cloud Foundation specifically — is the migration different?

VCF migrations are larger (more components, more integrations). The discovery phase covers SDDC Manager, Workspace ONE, Aria, NSX-T overlays, and any custom service definitions, and the assessment sizes the extra work.

What if we use vCloud Director for our customers?

vCD migration is the most common path for service providers. Tenant model maps to Cozystack Tenant CRD, service catalog maps to ApplicationDefinition, billing maps to the Ænix WHMCS integration or your own billing system. We are happy to walk through the architecture in a call.

Is GPU live migration supported?

No — industry-wide limitation, not Cozystack-specific. VMware vGPU live migration has known caveats too. For inference workloads we recommend designing for stateless restart rather than relying on GPU live migration.

The VMware migration market in 2026 is a different conversation than in 2022. Broadcom-induced exits have produced enough customer experience that the patterns that work and the patterns that fail are documented. This article covers the working version.

Migration paths — three options

There are three honest VMware migration paths in 2026:

Path 1: VMware-managed migration (vendor-tools-led)

Use Broadcom-supplied or Red Hat-supplied tooling to migrate within or out of the VMware ecosystem. Tools: VMware HCX (intra-VMware), Red Hat Migration Toolkit for Virtualization (MTV) for VMware → OpenShift, vendor-specific (Nutanix Move, Scale Computing tools).

Pros: Vendor support during migration; established tooling.

Cons: Locks into vendor relationship at destination. Limited choice of destination architecture.

Path 2: KubeVirt-based migration (open-source destination)

Convert VMware VMs to KubeVirt format on the destination platform (Cozystack, OpenShift Virtualization, vendor KubeVirt platforms). Tools: virtv2v (RedHat), forklift (RedHat), KubeVirt CDI (community), custom scripts.

Pros: Open destination architecture, no vendor lock-in at destination, modern Kubernetes-native foundation.

Cons: More integration work upfront; team learning curve.

Path 3: Lift-and-shift to public cloud

Move VMware workloads to VMware-on-cloud (VMware on AWS, Azure VMware Solution, Oracle VMware Solution). Defers the architectural question.

Pros: Fastest migration path; smallest re-architecture.

Cons: Doesn’t address Broadcom pricing pressure (often makes it worse); doesn’t address sovereignty; usually a stop-gap rather than a strategy.

For most 2026 migrations driven by Broadcom pricing or sovereignty, Path 2 (KubeVirt-based) wins. This article focuses there.

Tooling for KubeVirt-based migration

KubeVirt CDI (Containerized Data Importer)

Native KubeVirt tool. Imports disk images (qcow2, VMDK, VHD, ISO) into KubeVirt PersistentVolumeClaims. Free, open-source, works.

Strengths: Native to KubeVirt; supported by KubeVirt operators; works with various source formats. Limits: Disk-level migration; doesn’t handle VM metadata, networking, or post-migration cleanup automatically.

Forklift / Migration Toolkit for Virtualization (Red Hat)

Red Hat’s tool for vSphere → KubeVirt (OpenShift Virtualization) migration. Open-source; works on any KubeVirt platform with manual configuration.

Strengths: Bulk VM migration; metadata preservation; production-tested. Limits: Originally OpenShift-focused; some integration work for non-OpenShift destinations.

virtv2v (Red Hat)

Lower-level conversion tool. Operates on individual VM images; converts vSphere VMs to KVM-compatible format with VMware Tools cleanup.

Strengths: Mature, used by Forklift internally; handles Windows VM driver cleanup. Limits: Single-VM tool; orchestration is your problem.

Cozystack-specific migration tooling

KubeVirt CDI + dedicated migration scripts that Ænix has built and reused across customer deployments. Covers VM image conversion, multi-tenant placement, network mapping into Cilium policies.

Strengths: used by Ænix in production migrations; Cozystack-tenant-aware. Limits: delivered as part of an Ænix engagement, not as a standalone open-source tool.

Vendor / commercial tools

• Nutanix Move — for VMware → Nutanix AHV migrations
• Scale Computing tools — for VMware → Scale HC3
• Various commercial migration tools — Carbonite, Veeam, etc.

These work for their specific destination; not relevant for KubeVirt-based migration to open destinations.

Strategy for VMware migration

Step 1: Discovery and assessment

Before tools, agree on:

• Trigger — what’s pushing the migration? (Pricing, sovereignty, scale, AI.)
• Workload portfolio — count, OS mix, criticality, dependencies.
• Destination architecture — what you’re moving to.
• Timeline constraints — VCF subscription expirations.
• Team capacity — to run the destination after migration.

Output: structured assessment with workload classification (migrate-now / later / stay / re-platform).

Step 2: Destination platform engineered

The destination platform must be production-ready before migration starts. This is where many migrations fail: workloads move to a destination that’s been engineered as a PoC, not as a production platform.

For Cozystack-based destinations: 1-3 months destination-build before migration cohort 1.

Step 3: Cohort-based migration

Cohorts of 10-50 workloads at a time. Each cohort:

1. Pre-migration validation (workload state, dependencies)
2. Image conversion
3. Network and storage cutover
4. Post-migration validation
5. Parallel-run (VMware + destination both active)
6. Customer signoff
7. VMware-side decommission

Cohort cadence: 1-3 cohorts per month at steady state.

Step 4: VCF expiration-aligned sequencing

Workloads move when VCF commitments lapse. Moving early into a paid-for commitment costs money. Moving late risks expiration without prepared destination.

Step 5: Decommission

VMware-side decommission as cohorts complete. Hardware repurposed where applicable.

Where migrations stumble

Stumble 1: destination not ready

Workloads moved to destination that’s not actually production-ready. Operational issues blamed on migration.

Stumble 2: big-bang cutover attempted

Weekend “we’ll move it all” — rarely works at enterprise scale. Cohort approach is the proven pattern.

Stumble 3: data gravity ignored

50TB production database moves are not a weekend job. Cross-network movement, cutover windows, dual-write periods all need engineering.

Stumble 4: networking redesign skipped

Cilium ≠ NSX. Network policies, ingress patterns, service-to-service auth all need redesign.

Stumble 5: post-migration capacity underestimated

Migration completes; platform team thinks they’re done. Actually, post-migration operations is when the workload starts. Capacity for ongoing ops is non-negotiable.

Cost ranges

For planning:

• Assessment: Platform Readiness Assessment, fixed price, 14 days focused or 28 days full.
• Destination platform foundation: live in weeks once hardware is ready; up to 3 months with integrations, depending on scale.
• Migration cohort labor: 8-15 person-days per cohort of 10-50 VMs.
• Total elapsed: about 8-12 months for ~100 VMs and 18-24 months for ~1,000 VMs, including planning and migration waves; mid-size estates fall in between, depending on dependencies.

Compared to ongoing VCF subscription: most customer engagements show net positive after Year 2 even accounting for migration cost.

Three pressures

1. NIS2 scope — transport is an Annex I sector (essential or important entity depending on size); Article 21/23 apply to ICT
2. AI optimization — routing, demand forecasting, predictive maintenance
3. Edge compute density — depots, ports, terminals, vehicles

Three-tier architecture pattern

• HQ Cloud — TMS, fleet management, AI training, customer-facing
• Regional sites — operational centres, regional dispatch, AI inference
• Edge — depots, ports, terminals, on-vehicle compute

Cozystack at all three tiers, with OT boundary at edge for safety-critical systems (rail signalling, automated terminal operations).

NIS2 architecture controls for transport

Standard Article 21 + 23 mapping; transport-specific:

• Multi-modal data sovereignty (cross-border freight data)
• Sub-contractor visibility (logistics chains often have 5+ levels)
• BCP for kinetic disruption scenarios
• Air-gap for safety-critical OT

AI use cases in transport

• Route optimization (real-time + planning)
• Demand forecasting
• Predictive maintenance for fleet
• Last-mile optimization
• Customer-facing AI (delivery ETA, customer service)

Most workloads are sustained 24/7 inference where dedicated GPU economics fit.

How Ænix engages

Standard Platform Readiness Assessment with transport workstream emphasis.

The telco cloud conversation in 2026 sits at an unusual intersection of pressures that no other vertical faces simultaneously. NFV deployments from 2015-2020 are aging out. NIS2 obligations apply across the operator’s infrastructure. Sovereign-cloud branded products are commercially attractive (regional sovereignty positioning is hard for hyperscalers to replicate). Edge-compute demand for 5G/6G workloads is growing. AI for network operations (traffic prediction, anomaly detection, customer-facing assistants) is emerging.

The architectural answer is rarely “one platform for everything” — telcos have multiple operational domains, each with its own constraints. But the patterns that work across all of them have converged on Kubernetes-native multi-site platforms with strong multi-tenant isolation, sovereign operation, and edge-aware scheduling.

What modernization actually replaces

Most tier-1 European telcos in 2026 operate three or four parallel infrastructure environments:

1. Legacy NFV environment (2015-2020 vintage)

Typically built on VMware Cloud Foundation, OpenStack-based vendor distros (Red Hat OSP, Mirantis Cloud Platform, Wind River, etc.), or vendor-specific NFVI (Ericsson CEE, Nokia CloudBand). Hosts VNFs from network-equipment vendors with vendor-specific certification requirements.

Modernization driver: vendor distribution lifecycles (Red Hat OSP EOL, Mirantis transition), VMware Broadcom pricing, operational complexity that compounds with team turnover. Most tier-1 operators have parallel modernization programmes running.

2. IT cloud (separate from NFV)

For business workloads, customer-facing portals, billing, OSS/BSS. Typically a separate VMware estate or hyperscaler-region deployment. Same Broadcom pressure on VMware; sovereignty pressure on hyperscaler deployment.

3. Edge compute (5G/6G era)

Distributed edge sites at substations, central offices, MEC nodes. Smaller-footprint deployments with intermittent connectivity to core. Often built on different stack from core platform (legacy small- footprint orchestration).

4. AI / data lake / network analytics

Newer environment for traffic prediction, anomaly detection, customer- facing AI. Often hyperscaler-based today; under sovereignty pressure to move on-prem for sensitive workload patterns.

The single-platform vision (and where it works)

The architectural attraction of Cozystack-based modernization is a unified platform across all four environments. One operational model, one upgrade lifecycle, one observability stack, federated identity, GitOps-driven changes.

Where this works:

• IT cloud workloads — straightforward Cozystack-based deployment. Most tier-1 telco IT cloud modernizations follow this pattern.
• AI / data lake / analytics — Ænix AI Platform workload patterns; sovereign by default; multi-tenant for cross-BU access.
• Edge compute — Cozystack supports small-footprint edge deployments with federation to core. Standardised across sites.

Where it requires nuance:

• NFV environment specifically — VNF certification with vendor equipment is still vendor-managed. The Cozystack platform underneath can host KubeVirt-based VNFs, but the certification stack remains vendor-bound. Some VNFs are certified on specific OpenStack distributions and need a parallel modernization track.
• Critical SCADA-style network OAM — air-gapped network management systems with their own operational model. Cozystack can host them, but the OT-style operational discipline differs from IT cloud.

The practical pattern: Cozystack as the IT cloud platform + AI platform + edge platform; NFV gets its own modernization track with Cozystack-equivalent architecture where vendor certification allows.

Multi-tenancy for telco

Telco operators have layered multi-tenancy needs:

• Operational BU separation — fixed broadband, mobile, enterprise, consumer; each BU has different operational ownership.
• Customer-facing services — telco offers cloud capacity to its business customers (sovereign cloud product line); each customer is its own tenant.
• Internal vs external workloads — internal-only workloads (OAM, observability backends) versus customer-facing workloads (cloud product, customer portal).
• Sectoral / regulated tenants — telcos increasingly host regulated workloads (financial sector adjacencies, public-sector contracts) under sectoral tenant isolation.

Cozystack’s Tenant CRD model with nested tenants supports all four layers natively. A tier-1 telco typically operates 5-50 top-level tenants with hundreds to thousands of nested tenants.

Sovereignty positioning

For telcos, sovereignty is not just compliance — it’s a commercial differentiator. European customers increasingly view US-headquartered hyperscaler dependencies as structural risk. Telcos can offer sovereign cloud products in their region that hyperscaler-managed offerings cannot match on the substantive sovereignty criteria.

Cozystack-based architecture supports this commercially:

• Encryption with a customer-held passphrase — opt-in volume encryption at rest (LINSTOR and LUKS); the key-management process is designed with the telco, which provides operational support
• Air-gap option — for classified or restricted-egress use cases
• Open-source substrate — exit-readiness built in; telco doesn’t lock customers into vendor relationship
• EU jurisdiction — telco’s EU presence + Ænix EU contracting entity (AENIX s.r.o.)

For sovereign cloud commercial product lines, the Ænix Public Cloud Platform is the typical pairing — multi-region, multi-DC, service-catalog depth, brand-engineered customer portal.

Edge compute realities

5G/6G drove distributed compute requirements that NFV-era architecture didn’t anticipate. Modern edge compute for telco operations spans:

• MEC (Multi-access Edge Computing) nodes at base-station or central-office locations for low-latency customer workloads
• Distributed RAN intelligence — vRAN / O-RAN workloads with real-time constraints
• Customer-edge compute — telco-managed edge instances on customer premises (factories, smart-grid sites, transport hubs)
• Sectoral edge — telco-operated edge for sectoral customers (financial branch offices, healthcare facilities, retail)

Cozystack supports edge deployment with reduced footprint (~3-node clusters at edge sites typical) with federation to regional and core platforms. Same operational model; same observability; differentiated service catalog per tier.

AI for telco operations

The AI workload patterns at tier-1 telcos:

• Traffic prediction and capacity planning — 24/7 inference on network telemetry
• Anomaly detection — security and operational anomalies in real- time
• Customer-facing AI — chatbot, billing assistance, customer service routing
• Network optimization — routing, traffic engineering, energy efficiency
• Fraud detection — transaction anomaly detection, identity validation
• Sectoral AI services — telco-hosted AI capacity for sectoral customers (banking AI, healthcare AI, public-sector AI)

Sustained-utilisation workload profiles dominate; this is exactly the case where dedicated GPU economics beat hyperscaler. The Ænix AI Platform fits.

Phasing a tier-1 telco modernization

The cloud platform follows the operator-scale pattern: a 3-6 month pilot, then 9-18 months to full multi-region operation. Edge expansion and NFV modernization run as longer parallel tracks, paced by site roll-outs and vendor lifecycles. Typical phasing:

Phase 0 — Strategic engagement (start of the pilot)

Architecture review across all four environments. Commercial alignment on sovereign cloud product line, sectoral positioning, modernization sequencing. Sponsor and workstream-lead identification.

Phase 1 — IT cloud modernization

Ænix Private Cloud Platform / Public Cloud Platform deployed. Internal workloads migrated. Customer-facing portal launched for sovereign cloud product.

Phase 2 — AI / data lake (parallel)

Ænix AI Platform deployed. Network analytics workloads moved on-prem. Customer-facing AI services launched.

Phase 3 — Edge expansion (ongoing, paced by site roll-out)

Edge sites stood up at MEC / central-office / customer-edge locations. Federated identity and observability across.

Phase 4 — NFV modernization (parallel, paced by vendor lifecycles)

Vendor-certified VNF replatforming where allowed. Greenfield deployments of new VNFs on Cozystack-based architecture. Legacy NFV maintained in parallel until vendor lifecycle forces upgrade.

Phase 5+ depends on customer-facing product growth: regional expansion, sectoral SKUs, new service families.

When this fits a telco

Strong fit:

• Tier-1 or tier-2 European telecom operator
• Active legacy NFV modernization programme
• Strategic intent to offer sovereign cloud product line
• Multi-million-euro budget envelope across multi-year programme
• Senior executive sponsorship (CIO / CTO / Cloud BU lead)

Marginal fit:

• Smaller operators (regional, MVNO-style) that sell cloud — Ænix Public Cloud Platform at provider scale, from the published price list, rather than a full operator-scale programme
• Operators with deep OpenStack-based NFV investment that still works — modernization can wait for vendor lifecycle to force it

Where to dig deeper

• Telco industry page — commercial landing
• Public Cloud Platform product page — the product for telcos that sell cloud services
• Sovereign cloud builder services — for sovereign cloud product line builds
• Sovereign AI services — for AI workload patterns
• Public Cloud Platform build phasing — operator-scale build phasing
• Sovereign AI architecture decisions — seven decisions for sovereign AI

SRE — Site Reliability Engineering — is one of the most-adopted and most-misunderstood engineering disciplines of the past decade. Most mid-to-large engineering organisations claim to “do SRE.” Far fewer actually run the discipline as Google’s original SRE book describes it: software engineering applied to operations, with explicit error budgets, SLOs that affect prioritisation, and a hard ceiling on operational toil.

What SRE means, precisely

The original Google formulation has three load-bearing properties:

1. SRE is software engineering applied to operations

SREs are software engineers — they write code, they ship systems, they automate operations. Not a relabelled ops team with the same job and a fancier title. The “is your SRE actually writing code?” test is a real test; teams that fail it have an ops function, not an SRE function.

2. Error budgets affect prioritisation

The SLO defines the bar. The error budget is the difference between 100% and the SLO target. When the error budget burns down, the product team’s prioritisation shifts — feature work pauses, reliability work becomes the priority. When the error budget is healthy, feature work can proceed.

Crucially, this is a contract, not a suggestion. SLO breach has real consequences for product roadmap. Organisations that have SLOs but don’t let them affect roadmap have observability, not SRE practice.

3. Operational toil is capped (often at 50%)

The Google formulation: SREs cannot spend more than 50% of their time on operational toil (repetitive, manual, automatable work). The remainder must go to software engineering that reduces toil. This makes the function self-improving over time — toil decreases, engineering capacity increases.

Teams that exceed the toil cap structurally have an ops function masquerading as SRE. The discipline degrades over months until the function is indistinguishable from traditional ops.

Three engagement models

In 2026, three structurally different SRE models exist in production:

Model 1: embedded SRE

SREs sit inside product teams. The product team includes 2-5 engineers, one of whom holds the SRE function. Same team owns features and reliability; the SRE shapes the architecture and on-call from inside.

Strengths: alignment between SRE and product priorities, fast feedback, no cross-team friction.

Weaknesses: SRE-quality varies per team, depending on the embedded person. Centralised SRE expertise doesn’t compound. Hard to scale across many product teams without losing consistency.

Fits: 100-300-engineer organisations where each product team has clear ownership and ~1 SRE-skilled engineer is available per team.

Model 2: centralised SRE function

SREs sit in a separate function with their own reporting line. They consult with product teams, set reliability standards, run shared infrastructure components, and operate the critical-path services.

Strengths: SRE expertise compounds. Standards are consistent across teams. Critical-path services have dedicated reliability owners.

Weaknesses: cross-team friction when SRE recommendations conflict with product team priorities. Risk of SRE-as-blocker rather than SRE-as-enabler.

Fits: 300+-engineer organisations with multiple critical-path services and a clear platform-engineering function. Often paired with platform engineering as a sister function.

Model 3: hybrid (embedded + centralised)

Embedded SREs in product teams handle team-specific reliability. Centralised SRE function operates shared infrastructure, sets standards, escalates for cross-team incidents.

Strengths: gets benefits of both. Most common pattern at large organisations (Google, Netflix, large fintech).

Weaknesses: requires substantial scale to justify the dual investment. Below 500 engineers, the overhead is hard to amortise.

Fits: 500+-engineer organisations with multiple business units and substantial reliability-sensitive workload portfolio.

Where SRE shows up as theatre

Patterns we see in assessments:

Theatre 1: SLOs exist, but don’t affect roadmap

SLOs are documented. Dashboards show SLO compliance. Quarterly reviews mention SLO trends. But product team prioritisation proceeds independently of error-budget state. When SLO breach happens, the response is incident-specific firefighting, not a roadmap re-prioritisation.

This is observability with SLO labels, not SRE practice. The distinction matters for buying decisions: organisations in this state don’t need more dashboards, they need a different governance model.

Theatre 2: SRE title applied to traditional ops

Operations engineers get retitled “SRE” without a change in job content. They still spend 90% of their time on ticket queue. They write no code beyond shell scripts. They have no error-budget authority.

This is a label change, not a discipline shift. Often happens during DevOps-to-SRE transitions that didn’t get executive sponsor investment.

Theatre 3: error budgets defined but never invoked

Error budgets are calculated. Some dashboards show them. But no process exists for what happens when they burn down. Effectively the same as no error budget.

Fix: write down the explicit feature-pause protocol that activates on error budget burn. Get product leadership to sign off on it. Test it once with an artificial burn-down before assuming it works in production.

Theatre 4: incident post-mortems without action items

Post-mortems happen after incidents. They get written. They go into a folder. No action items are tracked to completion. The same incident class recurs within 6 months.

Fix: post-mortem action items go into the same backlog as feature work, with named owners, due dates, and explicit prioritisation. The post-mortem hasn’t worked if its action items don’t ship.

What Ænix SRE engagement delivers

We typically engage with organisations where SRE practice is at one of three states:

• Pre-SRE — no formal SRE function; reliability is incident- driven firefighting. Engagement covers function definition, hiring plan, initial SLO definition for critical services.
• Theatre-state SRE — SRE titles and dashboards exist, but discipline hasn’t landed. Engagement diagnoses which theatre patterns are operating, designs corrections, often involves cross-functional governance work.
• Mature SRE needing expansion — discipline works in one business unit, needs to scale across the organisation or pick up new service families (AI/GPU workloads, edge compute, sovereign cloud). Engagement focuses on consistency and expertise transfer.

Workstream 1 — SLO definition

For each critical service, define:

• SLI (Service Level Indicator) — what we measure
• SLO (Service Level Objective) — the target
• Error budget — the difference between 100% and the SLO
• Burn-down policy — what happens when the budget is being consumed
• Recovery threshold — what restores feature-work prioritisation

Ænix doesn’t define SLOs for you in isolation — we facilitate the workshop where engineering and product leadership co-author them. SLOs without joint ownership don’t stick.

Workstream 2 — Incident response process

Roles (incident commander, scribe, communicator). Severity classification. Runbook structure. Escalation paths. Blameless post-mortem template. Action-item tracking.

This is often the highest-leverage workstream — the framework multiplies the effectiveness of every future incident.

Workstream 3 — Toil measurement and reduction

Inventory current SRE / ops team work. Categorise: toil (repetitive, manual, automatable) versus engineering (durable, automation- generating). Measure toil ratio.

Recommend the 3-5 automations that reduce the most toil. Stage them by ROI. Hand off to engineering team for implementation; we support if needed.

Workstream 4 — Observability stack

Ænix’s default observability recommendation: VictoriaMetrics for metrics, VictoriaLogs for logs, OpenTelemetry for tracing where applicable. Self-hosted (sovereignty-friendly, lower-overhead than Prometheus + Loki at scale, no SaaS vendor data-residency leak).

For organisations already on a different stack, we work with what’s in place rather than push replacement. Observability stack matters; SRE discipline matters more.

Workstream 5 — Function design

Embedded / centralised / hybrid model recommendation per the engineering organisation’s profile. Headcount planning. Hiring priorities. Reporting line. Interface with platform engineering (if separate function) and product teams.

The Cozystack reliability defaults

For organisations running an Ænix platform product, SRE practice gets a head start because the platform ships with SRE-aligned defaults:

• Observability built in — VictoriaMetrics + VictoriaLogs pre-deployed, with alert rules for the platform components
• Declarative change history — tenants and services are Kubernetes resources managed through GitOps, so every change has a commit, an author and a review
• Backup-restore patterns — Velero + per-app PITR, with RPO / RTO documented per service during the engagement

SLO definitions and failure-injection (chaos) tooling are not shipped as platform features; they are designed with you in the engagement.

This lets the SRE engagement focus on the organisation-specific work (function design, SLOs aligned to business priorities, governance) rather than rebuilding the technical substrate.

When this engagement fits

Strong fit:

• Engineering organisation 200+ engineers with reliability becoming a board-level concern
• Recent incident pattern that exposed reliability gaps
• Regulator-driven RTO/RPO obligations (DORA Articles 11-12, NIS2 Article 21(2)(c))
• Existing observability investment but no clear SRE discipline
• Platform engineering function exists or is being built (SRE pairs naturally with platform engineering)

Marginal fit:

• Smaller organisations (<100 engineers) — usually embedded SRE rather than separate function; Ænix engagement can be lighter (workshop + advisory rather than full multi-month engagement)
• Organisations with mature SRE in one BU needing expansion — scope can be narrower

Poor fit:

• Pure firefighting culture without engineering leadership sponsorship for the discipline shift — SRE engagement without executive backing degrades to incident response training, which is helpful but not what we sell

Where to dig deeper

• SRE consulting services — the commercial landing
• DevOps best practices for 2026 — the eight DevOps practices including SRE
• Platform engineering vs DevOps vs SRE — terminology and function design
• Cloud engineering disciplines — the seven disciplines that compound

Editor’s note (September 2026): this release announcement is preserved as it was published on May 28, 2026. Cozystack has moved on since — the current release line is 1.6. For the latest patch release, see the Cozystack releases and the current documentation. Documentation links below deliberately point at the v1.4 docs, which describe the release as shipped.

Cozystack v1.4.0 is now available. The release was published on May 19, 2026, and rolls up every fix shipped in the v1.3.1 to v1.3.3 patch line.

This cycle is focused on the operational experience of running Cozystack as a production platform: a faster dashboard architecture, more durable tenant Kubernetes workers, clearer resource sizing, backup workflows for managed applications, better GPU utilization, safer ingress publishing, and fewer race conditions during first installs and upgrades.

Main highlights

New schema-driven dashboard UI

Cozystack 1.4 ships a rewritten dashboard from the cozystack/cozystack-ui project. The old openapi-ui plus BFF stack has been replaced by a React 19 and TypeScript frontend that talks directly to the Kubernetes API.

The new architecture removes an extra process and proxy layer while keeping the dashboard schema-driven. It also improves several day-to-day workflows:

• VNC access for virtual machines now uses dynamic WebSocket URLs instead of deployment-specific localhost assumptions.
• The dashboard can read ApplicationDefinition resources for the application catalog and marketplace.
• Operators can inject runtime branding through a ConfigMap, including logos, names, and brand colors, without rebuilding the image.
• Existing /openapi-ui/* bookmarks are redirected to the new console.
• The package has been renamed consistently to cozy-dashboard.

The new dashboard exposes the IaaS marketplace driven by ApplicationDefinition resources.

The PaaS catalog covers managed databases, messaging, object storage, secrets, search, and inference services.

Deploying a managed Kubernetes cluster uses the same schema-driven form, with cluster addons exposed declaratively.

A managed HTTP cache deployment, with PVC sizing, storage class, endpoints, and resource parameters all generated from the application schema.

Documentation:

• Deploying applications through the new dashboard
• ApplicationDefinition reference
• White-labeling and runtime branding

Persistent worker-node storage for tenant Kubernetes

Tenant Kubernetes worker VMs now use PVC-backed persistent disks through KubeVirt dataVolumeTemplates. Previously, workers used ephemeral emptyDisk storage, which meant kubelet certificates, kubeconfig, and containerd state were lost after a VM reboot. A rebooted worker could lose its identity and require manual recovery.

In v1.4, worker state survives VM restarts. The NodeGroup field ephemeralStorage is renamed to diskSize, and a new per-nodeGroup storageClass option lets operators control where worker disks are provisioned. Migration 39 rewrites legacy values automatically during upgrade.

Existing tenant clusters will roll worker nodes once because the KubeVirt machine template changes. Operators should plan capacity for the rollout and choose the storage class intentionally. For many worker-node disk scenarios, the local StorageClass is recommended because worker disks now survive restarts and do not need DRBD replication semantics.

Documentation: Tenant Kubernetes configuration.

Instance-type resource presets

Resource presets now follow a cloud-style <series>.<size> taxonomy. The new model covers five CPU-to-memory series:

• t1 for tiny and low-memory workloads.
• c1 for compute-balanced workloads.
• s1 for standard services such as proxies and caches.
• u1 for universal workloads such as databases and messaging.
• m1 for memory-heavy workloads such as search and analytics.

Each series includes eight sizes from nano to 4xlarge, giving operators and tenants 40 presets in total.

The previous flat names such as small, medium, and large are still accepted as deprecated aliases. Existing deployments keep the same CPU and memory values, while Migration 39 rewrites stored values to the new names. The Cozystack API now emits deprecation warnings when app CRs still use legacy preset names.

Documentation: Resource presets.

Declarative backup strategies for managed applications

The backup strategy controller now supports PostgreSQL, MariaDB, ClickHouse, and FoundationDB. Tenants can define a strategy together with BackupClass, Plan, BackupJob, and RestoreJob resources, while the controller composes the backend-specific objects for each managed service.

The new strategies support scheduled backups, ad-hoc snapshots, in-place restores, and restore-to-copy workflows against S3-compatible object storage. Credentials are referenced through Kubernetes Secrets instead of being stored inline, and controller RBAC is constrained so it can only access explicitly referenced secrets.

This extends the existing backup flows for VMInstance and VMDisk and moves Cozystack closer to full backup coverage across the managed application catalog.

Documentation:

• Managed app backup configuration
• Application backup and recovery

HAMi-based fractional GPU sharing

Cozystack 1.4 adds hami as an optional system package. HAMi v2.8.1, a CNCF Sandbox project, provides fractional GPU sharing for tenant Kubernetes clusters.

With HAMi enabled, tenant workloads can request resources such as nvidia.com/gpu, nvidia.com/gpumem, and nvidia.com/gpucores, allowing multiple pods to share one physical NVIDIA GPU with explicit memory and compute slicing. The integration includes the device plugin, scheduler extender, mutating webhook, and RuntimeClass. It is exposed through an opt-in hami.enabled toggle and depends on the NVIDIA GPU Operator.

There is one important compatibility note: HAMi compute isolation depends on container images with glibc older than 2.34. Memory enforcement works broadly, but Alpine and musl-based images are not supported for HAMi-core compute isolation.

Documentation: GPU sharing with HAMi.

One switch for PROXY protocol and hairpin NAT

The new publishing.proxyProtocol: true option enables PROXY protocol on the host ingress-nginx and deploys Ouroboros to solve the related hairpin-NAT problem.

When PROXY protocol is enabled, in-cluster traffic to the cluster’s own public hostnames can otherwise reach ingress-nginx without the required PROXY header. Ouroboros fixes that path through CoreDNS rewrite snippets. Cozystack exposes it as both a host-level system package and a per-tenant addon through addons.ouroboros.enabled.

The default behavior is unchanged. Clusters that do not enable PROXY protocol do not receive new resources.

Documentation: PROXY protocol and hairpin NAT.

Better HelmRelease behavior and tenant bootstrap reliability

The Cozystack operator now exposes HelmRelease generation knobs as operator flags and chart values, including interval, retry interval, install timeout, upgrade timeout, and max history.

The retry strategy now uses RetryOnFailure, which avoids uninstall-and-reinstall loops when a first install is slow. Applications can also set a per-Application install and upgrade timeout through the release.cozystack.io/helm-install-timeout annotation. Tenant Kubernetes uses this to give Kamaji enough time during cold bootstrap, fixing the recurring wait hr/tenant-kubernetes timeout failure mode.

Documentation:

• Tenant Kubernetes operations
• Troubleshooting Flux CD

Kubelet reservations for worker nodes

Tenant Kubernetes worker nodes now receive automatically computed kubelet reservations for CPU and memory. This keeps kubelet itself from being targeted under memory pressure and makes scheduler and autoscaler decisions more accurate.

Cluster-autoscaler annotations now report allocatable CPU and memory instead of raw totals, so autoscaling decisions match what Kubernetes can actually schedule.

Documentation: Tenant Kubernetes operations.

Also in v1.4.0

• PostgreSQL parameters are now typed and protected by a denylist for dangerous values such as archive_command, restore_command, ssl_passphrase_command, dynamic_library_path, and *_preload_libraries.
• Keycloak gains extraEnv and user profile customization support.
• The etcd application exposes S3 backup schedules through the updated etcd-operator.
• Per-package upgradeCRDs policy is now configurable.
• cozyreport now collects Flux, cert-manager, host context, application resources, and a top-level summary.txt.
• SeaweedFS tenant ingress limits single PUT requests to 5 GB.
• GPU observability dashboards and recording rules were added for Grafana and VictoriaMetrics.
• VMInstance port filtering is fixed under the new cozy-proxy v0.3.0 mode.
• LINSTOR CSI is updated with fixes for dual-attach and transient demotion errors.

Documentation:

• PostgreSQL configuration
• Keycloak and OIDC
• etcd service configuration
• SeaweedFS service configuration
• Monitoring dashboards
• Troubleshooting and diagnostics

Platform components

Cozystack 1.4 updates the platform base and several core packages:

• Talos: v1.12.7 to v1.13.0
• cert-manager: v1.19.3 to v1.20.2
• Cilium: v1.19.1 to v1.19.3
• NVIDIA GPU Operator: v25.3.0 to v26.3.1
• etcd-operator: v0.4.2 to v0.4.3
• KubeVirt: v1.6.3 to v1.8.2
• cozy-proxy: v0.2.0 to v0.3.0
• linstor-csi: v1.10.6
• HAMi: v2.8.1
• Ouroboros: v0.7.2

Documentation:

• Platform stack overview
• Upgrade guide

Upgrade notes

Most operators can upgrade to v1.4.0 without manual configuration changes. Cozystack keeps the same API surface for existing workloads, and in-platform migrations handle the main value rewrites.

There are several operational details to plan for:

• Tenant Kubernetes workers will roll once. The ephemeralStorage to diskSize migration is automatic, but existing worker VMs are replaced one by one because the KubeVirt machine template changes.
• KubeVirt VMs that were already running before the platform upgrade need a cold restart after upgrading. The KubeVirt jump from v1.6.3 to v1.8.2 crosses an upstream QEMU change, and live migration of pre-upgrade VMs can fail. New VMs created after the upgrade are unaffected.
• Legacy resource preset names still work as deprecated aliases, but new deployments should use the <series>.<size> names.
• PostgreSQL deployments that use denylisted parameters will fail to render until those parameters are removed.
• cert-manager v1.20 changes the default container UID/GID to 65532. Operators with custom PodSecurityPolicy, imagePullSecrets, or filesystem-mounted certificates pinned to the previous UID should review their configuration.

Documentation:

• Upgrade guide
• Tenant Kubernetes operations
• Virtualization operations
• Resource management
• PostgreSQL configuration

Documentation worth knowing about

• New dashboard and application catalog
• ApplicationDefinition reference
• White-labeling and runtime branding
• Tenant Kubernetes configuration
• Resource presets
• Managed app backup configuration
• Application backup and recovery
• GPU sharing with HAMi
• PROXY protocol and hairpin NAT
• Upgrade guide
• Cozystack v1.4 documentation

Thank you to all contributors

This release was shaped by the work of @androndo, @Arsolitt, @dislogical, @dvc, @IvanHunters, @kvaps, @lexfrei, @matthieu-robin, @mattia-eleuteri, @myasnikovdaniil, @sircthulhu, and @tym83.

A special welcome to first-time contributors @dvc and @dislogical. Thank you all.

Release links

• Cozystack v1.4.0 on GitHub
• Full changelog v1.3.0 to v1.4.0
• Cozystack UI
• HAMi
• Ouroboros

Join the community

• GitHub: cozystack/cozystack
• Telegram: @cozystack
• Slack: #cozystack on the Kubernetes workspace (invite)
• Subscribe to our community meetings calendar
• Add meetings to your calendar

Cozystack 1.4: New Dashboard UI, Persistent Tenant Workers, Backup Strategies, and Fractional GPU Sharing was originally published in Ænix on Medium, where people are continuing the conversation by highlighting and responding to this story.

The seven decisions

1. Trigger profile

What’s pushing toward sovereign? (Regulated data class / inference economics / auditability / air-gap requirement)

2. Regulatory scope

Which regulators bind you? (DORA, NIS2, sectoral, sovereign-cloud mandate, GDPR cross-border)

3. Model selection

Open-weight vs proprietary. Common 2026 open-weight: Llama, Mistral, Qwen, DeepSeek, Phi, Gemma. Choice depends on:

• Language requirement (multilingual vs English)
• Workload type (chat / RAG / code / vision / embedding)
• Licence (commercial use, attribution, redistribution)
• Capability target

4. Hardware sizing

• NVIDIA H100/H200 — workhorse for fine-tuning and high-throughput inference
• NVIDIA Blackwell — newer, highest memory bandwidth
• NVIDIA L40S — 48GB, multi-tenant inference fleet
• NVIDIA A100 — cost-effective, second-hand market
• AMD MI300/MI325 — alternative when ecosystem fits

5. Multi-tenancy model

• Single-tenant: lab / single-team / PoC
• Multi-tenant via Tenant CRD: enterprise platform / customer-facing
• Cluster-per-tenant: highest isolation, operationally expensive

6. Sovereignty controls

• Encryption keys customer-controlled (HSM)
• Supplier transparency to second hop
• Audit-trail completeness in regulator-consumable formats
• Air-gap option for most sensitive workloads

7. Operational model

• Customer-operated (you run it)
• Vendor-operated (Ænix or similar runs it)
• Hybrid (you operate; vendor 2nd-line)

How decisions interlock

The seven aren’t independent. Trigger profile shapes regulatory scope; regulatory scope shapes sovereignty controls; sovereignty controls shape operational model; operational model affects model selection feasibility.

Common combinations

Pattern 1: Regulated finance + sustained inference + multi-tenant DORA + Article 28 controls + multi-tenant Tenant CRD + opt-in volume encryption with a customer-held passphrase + Ænix-managed operations + open-weight (Llama 70B class) on H100/L40S fleet.

Pattern 2: Public sector + air-gapped + classified data Sovereign-cloud mandate + air-gap + customer-operated + open-weight (Llama / Phi) on customer hardware.

Pattern 3: AI startup + sustained 24/7 inference + customer-facing No specific regulator + cost economics trigger + multi-tenant + customer-operated + open-weight + GPU mix matched to workload.

How to use the decision guide

Answer the questions above in order and note your answers; the architecture options narrow naturally. The sovereign AI decision guide walks through the same decisions in more depth.

For specific engagement see Sovereign AI services.

The energy sector’s infrastructure-modernization conversation in 2026 sits at an unusual intersection: NIS2 compliance pressure, AI-driven grid-optimization demand, IT/OT convergence reality, edge compute requirements at substation density, and the irreducible operational fact that grid hardware refresh cycles are measured in decades. Few other sectors face this combination simultaneously.

Three pressures converging on energy infrastructure

Pressure 1: NIS2 compliance with operational reality

NIS2 Article 21 risk-management measures apply to energy operators (Annex I sector; essential or important entity depending on size). Article 23 incident reporting (24-hour / 72-hour / 1-month timelines) requires telemetry tuned for security, not just performance. Article 21(2)(d) supply-chain security requires mapping ICT third parties to the second hop at minimum.

For energy operators with legacy SCADA + DCS + GIS + energy-management systems integrated through years of one-off engineering, Article 21’s “documented risk register per critical workload” is non-trivial.

Pressure 2: AI for grid operations

Grid forecasting (load, generation, weather impact), demand response automation, predictive maintenance (transformer / line / substation health), market-price optimization — all increasingly use ML.

The economics drive sustained workloads (24/7 model serving) where dedicated GPU infrastructure usually wins over hyperscaler GPU. Combined with grid-operational data sensitivity (often legally protected), sovereign AI is the natural answer.

Pressure 3: Edge compute at substation density

Modern grid operations distribute compute toward generation, distribution, and storage edges. Substations (10s-100s per operator), distributed generation sites, microgrids, EV charging clusters — each may host workload that needs local compute with intermittent central connectivity.

Traditional centralized SCADA architecture doesn’t scale to this density. Modern smart-grid architecture distributes the platform.

Smart-grid architectural reference

Bring the three pressures together and the architecture looks roughly like this:

HQ Cloud (Cozystack)
   ├── Energy management system (EMS)
   ├── Grid analytics + AI training
   ├── Forecasting models (load, generation, weather)
   ├── Market integration (ENTSO-E, market operators)
   ├── Customer billing + customer-facing portals
   └── Compliance + audit log aggregation
        ↓
Regional control sites (Cozystack)
   ├── SCADA aggregation
   ├── Regional grid management
   ├── AI inference (forecasting models)
   └── Disaster recovery
        ↓
Substation edge (Cozystack)
   ├── Local SCADA / RTU integration
   ├── Real-time grid-edge compute
   ├── IoT data ingestion (sensors, smart meters)
   └── Air-gapped OT boundary

Cozystack runs at all three layers under one operational model, with policy and identity federated, but the OT/IT boundary at substation tier is intentionally restricted.

IT/OT convergence — boundaries that work

The naive “merge IT and OT” pattern is dangerous. The working pattern in 2026 is convergence with boundaries:

Strict boundary: OT zone

• Substation OT systems (SCADA, RTU, IEDs) live in OT zone
• Air-gapped or strictly restricted-egress to IT zone
• Updates via controlled channels (Harbor mirror, manual approval)
• Cybersecurity tooling is OT-aware (not generic IT EDR)
• Identity model separate from workforce IT

Permeable boundary: IT zone

• Grid analytics, AI inference, forecasting, customer-facing apps live in IT zone
• Standard cloud-native operational model
• IT cybersecurity tools and identity federation
• Cloud-native observability + compliance tooling

Bridge: data fabric

• Carefully-controlled data flows from OT to IT (typically read-only or one-way)
• Data validation + sanitization at boundary
• Audit-trail completeness at boundary

The architectural pattern: Cozystack runs the IT zone and the bridge layer. OT zone is its own dedicated infrastructure with controlled egress. Both can be deployed on Cozystack but with strict policy boundaries.

Cozystack architectural advantages for energy

1. Multi-site under one operational model

Cozystack platforms federate across central + regional + substation tiers. Single operational model, single platform team, consistent observability, GitOps-driven changes.

2. Air-gap support for OT

Documented air-gap install workflow. Suitable for OT zones that cannot have internet egress. Updates via Harbor mirror or controlled channels.

3. AI infrastructure native

KubeVirt for legacy AI workloads, native Kubernetes for modern ML pipelines. Passthrough of whole GPUs or NVIDIA vGPU (requires your NVIDIA vGPU licence) for VM-bound workloads, and HAMi fractional sharing (GPU memory and compute cores) for containers sharing GPUs across forecasting models. NVIDIA data-centre GPUs are supported through the NVIDIA GPU Operator; MIG and time-slicing are on the roadmap.

4. Multi-tenant for cross-BU

Tenant CRD model accommodates generation / transmission / distribution / retail BUs with separate isolation. For unbundled markets, this is non-optional.

5. Sovereign by architecture

Open-source platform on customer-controlled hardware. Opt-in volume encryption at rest (LINSTOR and LUKS) with a passphrase you hold. Audit-trail completeness in regulator-consumable formats. NIS2-aligned without bolt-on workarounds.

6. Long operational horizon

Apache 2.0 licence + CNCF Project community governance fits decade-plus grid operational planning. Vendor roadmap risk is minimized.

NIS2-specific architecture controls

For each Article 21 sub-requirement that touches infrastructure:

• Risk register per critical function — workload-level mapping, including OT-zone boundaries
• Incident detection at 24-hour timeline — telemetry tuned for security, not just performance
• Business continuity tested — annual exercises with documented results, including substation-tier failover
• Supply-chain transparency to second hop — every ICT vendor + sub-contractor mapped
• MFA for privileged accounts — universal in IT zone; appropriate substitutes in OT zone (smart-card, dedicated KVM)
• Cryptography with customer-controlled keys — HSM-based for sensitive grid-operational data
• Vulnerability management with CVE response SLA — challenging in OT zone (long maintenance windows); architecture must support staged remediation

AI workloads on grid-operational data

Common AI use cases at energy operators:

• Load forecasting — short-term (1-24 hours), medium-term (1-30 days), long-term (1-5 years)
• Generation forecasting — especially for renewables (wind, solar) where weather is decisive
• Predictive maintenance — transformer health, line condition, substation equipment
• Demand response automation — for flexible loads (industrial customers, EV charging, storage)
• Market price optimization — for participating in wholesale markets
• Grid topology optimization — power flow analysis, congestion management
• Customer-facing AI — chatbot, billing analysis, energy-advice AI

The data feeding these models is often legally protected (customer data, grid-operational data, market-sensitive data). Sovereign AI infrastructure on customer hardware is the natural deployment pattern.

Typical hardware sizing for mid-size energy operator (5-10 GW generation portfolio): 16-64 GPUs across H100/L40S, with elastic burst capacity for re-training cycles.

Migration patterns from legacy

Most energy-sector smart-grid platforms in 2026 are migrations from legacy mixes of:

• Vendor-specific SCADA platforms (often heavily customized)
• Legacy virtualization (VMware-heavy)
• Standalone forecasting systems (commercial software)
• Spreadsheet-based grid analytics (still surprisingly common)

Migration sequencing:

1. Discovery — workload inventory, OT/IT zone mapping, regulatory scope
2. Cozystack foundation — central tier first
3. Regional tier rollout — cohort by cohort
4. AI workload migration — from legacy forecasting to Kubernetes-native
5. Edge tier rollout — substation-by-substation, slowest tier due to operational risk
6. Legacy decommission — staged as cohorts complete

For a mid-size operator, the multi-site platform follows the operator-scale pattern: a 3-6 month pilot on the central tier, then 9-18 months to full multi-site operation. The substation roll-out can run longer, paced by maintenance windows.

Common pitfalls

Pitfall 1: weak OT/IT boundary

“We’ll converge OT and IT” without strict boundary controls produces operational risk. The cybersecurity model in OT is different; the boundary must be deliberate.

Pitfall 2: hyperscaler-only AI

Putting grid-operational AI in hyperscaler region. Often incompatible with NIS2 supplier-concentration requirements; sometimes legally problematic for customer-data-driven workloads.

Pitfall 3: substation-tier underspecified

Treating substation tier as “just SCADA” misses that modern smart-grid architecture distributes compute to substations. Needs platform-engineering treatment.

Pitfall 4: skipping NIS2 architecture during platform build

“We’ll add NIS2 controls in v2” produces retrofit cost that exceeds doing it from start.

Pitfall 5: under-engineered DR for grid criticality

Grid operations have public-safety implications. DR posture must be tested and proven, not documented only.

Where Ænix engages

The standard Platform Readiness Assessment with energy-specific workstream emphasis covers the full picture. For details and engagement structure see energy industry page.

For specific triggers see NIS2 compliance, Sovereign AI, Data sovereignty.

Want to dig deeper?

• Energy industry page — engagement details
• NIS2 compliance — essential-entity regulatory
• Sovereign AI — AI on grid data
• Platform Readiness Assessment — methodology
• Cozystack — open-source platform foundation

Most coverage of cloud repatriation is either ideological (“public cloud was always too expensive”) or vendor-led (“buy our private-cloud appliance”). Neither helps the platform engineer or infrastructure lead who has to translate a board-level decision into running systems. The work below is what we actually do during an Ænix repatriation engagement.

Why repatriation is happening now

Three independent pressures hit the same architectures at the same time:

Cost cliff after the renewal cycle. Public-cloud spend that looked acceptable in 2020-2022 has compounded for several years. Renewal cycles are now meeting boards that did not approve the trajectory. Reservation under-utilization, egress costs, and “we’ll right-size later” technical debt are all visible.

Regulatory pressure. DORA in force since January 2025; NIS2 transposition across EU member states; sectoral rules sharpening in financial services, public sector, healthcare; and explicit sovereign-cloud mandates in Kazakhstan, France, Germany, and other markets. Many critical-function workloads can no longer comfortably run in a single hyperscaler region under a US-headquartered provider.

AI and inference economics. GenAI training and inference at scale have egress, GPU-pricing, and data-residency profiles that hyperscalers were not designed to optimize. Some workloads make sense in hyperscaler GPU; many do not.

The combination has shifted repatriation from “what some hipsters do” to a normal part of cloud strategy. Done well, it produces 30-60% cost reduction on the workloads that move, plus regulator-aligned architecture. Done badly, it produces an under-engineered on-prem environment that combines the worst of both worlds.

Repatriation isn’t all-or-nothing

The single most common mistake at the strategy level is treating repatriation as a binary decision. It almost never is.

A typical repatriated estate, after a year of work, looks like:

• 30-50% on-prem or private cloud — steady-state workloads, regulated workloads, expensive workloads, latency-critical workloads
• 30-50% remaining in public cloud — elastic / spike workloads, hyperscaler-proprietary services with no realistic alternative, latency-sensitive customer-facing workloads where hyperscaler edge is decisive, very small workloads where the operational cost of repatriation exceeds the savings
• 10-20% in transition — workloads being moved, in PoC, or under reassessment

The workstream that classifies workloads — repatriate now / repatriate later / keep in cloud — is the most consequential single deliverable of a repatriation engagement.

Five-step playbook

Here is the sequence of work in a structured reverse cloud migration. Each step has a defined deliverable and a defined precondition for moving to the next step.

Step 1 — honest TCO modelling

Most organizations do not actually know their real public-cloud TCO. The bill is one number; the real cost includes:

• Egress charges, especially for backup, observability, and cross-region traffic
• Reservation / commitment under-utilization
• Idle and over-sized resources never reclaimed
• Hyperscaler-managed services priced at a premium over self-managed equivalents
• Hidden cost of vendor-lock-in: switching cost when something fails
• Cost of platform-engineering capacity dedicated to managing public-cloud-specific complexity

A honest TCO model captures all of these and compares them to a realistic destination-cost model that includes:

• Hardware acquisition and refresh cost over 5 years
• Datacenter or colocation cost
• Network bandwidth, including egress between sites
• Storage tiering and growth
• Backup and DR infrastructure
• Identity, observability, and platform tooling
• Platform-engineering capacity needed to operate the destination
• Software licences where applicable

The honest model usually shows on-prem economics 30-60% better for steady-state workloads, and 0-20% worse for highly elastic workloads. The interesting question is which workloads are which.

Deliverable: TCO model in a spreadsheet your CFO can audit. Public-cloud-current-state vs. destination-target-state, sensitive to occupancy assumptions.

Step 2 — workload classification

Every workload gets one of four labels:

• Repatriate now — clear cost or regulatory case, low migration friction, no hyperscaler-only dependencies.
• Repatriate later — case is clear but commitments not yet expired, or destination architecture not ready.
• Stay in cloud — workload is genuinely better-suited to hyperscaler economics or hyperscaler-only services.
• Reassess — not enough data to decide; needs PoC or instrumentation.

The classification considers:

• Compute pattern (steady vs. spiky)
• Data gravity (size, growth rate, regulatory class)
• Hyperscaler-proprietary dependencies
• Network adjacency (does it need to be next to other workloads?)
• Latency sensitivity
• Available commitment / reservation expiration
• Migration effort estimate

Deliverable: workload table with classification, ranked by net repatriation ROI. The top-10 list usually accounts for 60-80% of the cost case.

Step 3 — destination architecture

This is where most repatriations go wrong: treating “the destination” as “an on-prem cluster” without engineering the platform underneath.

A real destination architecture has answers for all of:

• Compute: virtualization platform (KubeVirt, KVM/libvirt, Hyper-V, VMware), hardware specs, lifecycle management, live migration, snapshots
• Storage: primary storage (LINSTOR / Ceph / SAN / vendor HCI), backup, DR, snapshots, replication topology
• Network: datacenter fabric, software-defined networking (Cilium, NSX equivalent), load balancing, BGP / routing, edge connectivity to remaining cloud workloads
• Identity: how IAM works, including federation with workforce identity
• Observability: metrics (VictoriaMetrics, Prometheus), logs (VictoriaLogs, Loki), tracing
• Backup and DR: Velero, restic, cross-site replication, RPO/RTO targets
• Platform-engineering function: the team that operates this; size, structure, on-call model
• Self-service surface: how product teams provision environments without filing tickets

A Kubernetes-native architecture (KubeVirt + Cilium + LINSTOR + Velero, or equivalent) is increasingly the default for repatriation projects in 2026 because it gives a coherent answer to all of the above without integrating ten unrelated vendors. Cozystack is one such platform; OpenStack, OpenShift Virtualization, and several vendor-led options are alternatives.

Deliverable: target architecture document. Diagram + named components + sizing + cost.

Step 4 — cutover sequencing

The cutover plan respects three constraints:

1. Commitment expirations. Workloads move when AWS RI / Azure RI / Savings Plan expirations open the economic case. Moving early into the lockup costs money.
2. Data gravity. Workloads that share data move together, or with explicit cross-cloud data flow during transition.
3. Risk concentration. No single cohort moves more than your operational capacity to manage rollback.

A typical 100-VM repatriation runs as 4-8 cohorts within a programme of about 8-12 months. Each cohort is parallel-run in cloud and on-prem until validated by the application owner.

Deliverable: cutover plan with cohort definitions, dates, dependencies, rollback paths, and signoff criteria.

Step 5 — operate the platform

The post-repatriation steady state is where the strategic gain materializes — if the platform was actually engineered. The post-repatriation team needs:

• Runbooks for the platform stack
• Monitoring and alerting that catches issues before users do
• Self-service paths for product teams (golden paths, IaC-based environment provisioning, GitOps-based deployment)
• A platform-engineering function with the right headcount to maintain pace

The most common failure mode at this stage is a successful migration followed by under-staffing of the platform team. Repatriation with a 5-person platform team responsible for everything that AWS used to handle is not sustainable.

Deliverable: operating model, headcount plan, runbook library, and a 12-month platform-engineering roadmap.

Common pitfalls

Beyond “TCO is wishful” and “destination is left for later,” four more pitfalls recur:

Pitfall 1 — treating repatriation as a cost project rather than a platform project

Repatriation that’s measured purely on cost reduction tends to deprioritize the platform work that makes the cost reduction sustainable. Two years in, the team has saved money but lost velocity, and the result is a partial reverse-repatriation back into hyperscalers.

The fix: repatriation goals include platform engineering maturity alongside cost. A successful repatriation produces a self-service platform whose time-to-environment is at least as good as what you had on hyperscalers — usually better.

Pitfall 2 — underestimating data gravity

Moving 50 TB of production data is not a weekend job. Cross-network movement, cutover windows, dual-write periods, rollback paths, and backup-during-migration all need explicit engineering. Teams that don’t plan for this end up in a multi-week emergency in the middle of the migration.

Pitfall 3 — buying a vendor-led “repatriation appliance”

Several vendors sell “private cloud in a box” as a repatriation answer. These work for a narrow class of customers but rebuild the lock-in problem with a different vendor. The vendor’s roadmap becomes your roadmap; the vendor’s support availability becomes your operational ceiling. Open-source platforms (Cozystack, OpenStack, OpenShift) avoid this trap.

Pitfall 4 — ignoring the human side

Platform-team morale, product-team relationships, and internal communications all shift during a repatriation. Engineers who built deep AWS skills may not see private cloud as a career step. Product teams accustomed to AWS Console may push back against new self-service paths. A repatriation program that doesn’t account for the human transition stalls in month 6.

When NOT to repatriate

Repatriation is the wrong answer when:

• You have a small IT team running a handful of services. Public cloud’s operational simplicity is genuinely better than what you’d build.
• Your workload portfolio is dominated by hyperscaler-proprietary services (e.g., heavy use of AWS Lambda + DynamoDB + Kinesis with no realistic alternatives).
• Your business is fundamentally elastic — traffic patterns where 10× spike capacity is needed for hours per day. Hyperscaler economics are good for this.
• Your platform-engineering capacity is already stretched beyond capacity. Adding repatriation to an under-resourced team makes both worse.
• Your renewal cycle is at year 1 of a 5-year commitment with steep penalties. Wait for commitment expirations.

A good repatriation engagement is honest about these cases. The Ænix engagement specifically does not push repatriation when staying in cloud is the right answer.

What about hybrid?

Most repatriated estates end up hybrid — a useful term, often vague in practice. There are three coherent patterns:

• Steady-state on-prem, elastic in cloud. Predictable workloads run on private platform; spike capacity overflows to cloud. Operationally complex; usually worth it for workloads with strong elastic patterns.
• Critical on-prem, non-critical in cloud. Regulated and core workloads on private cloud; auxiliary workloads (analytics, internal tooling, dev/test) in public cloud. Operationally simpler; common for financial-services repatriations.
• Geographic split. EU workloads on-prem in EU; non-EU workloads in regional public cloud. Driven by sovereignty rather than cost.

The hybrid pattern that fits depends on the trigger that drove repatriation — cost, regulator, or sovereignty.

How long does this take?

A typical repatriation, end-to-end:

• 14 or 28 days: assessment phase (Platform Readiness Assessment, fixed price, with cost-and-cloud-spend workstream emphasis).
• 3-12 months, overlapping with the first cohorts: destination platform build — greenfield infrastructure, base services, observability, identity, IaC and GitOps tooling, runbooks. The platform is usable for the first cohort well before the build is complete.
• The bulk of the programme: workload migration in cohorts. Earliest cohorts move quickly; later cohorts respect commitment ladders. For ~1,000 VMs, expect 18-24 months in total.
• Ongoing: platform operation and continuous optimization. The post-repatriation platform is a long-term asset that compounds value.

For an organization with 100 VMs and a moderate cloud bill, total elapsed time from “we should look at this” to “we are running on the destination platform” is about 8-12 months.

Where to start

If repatriation is on the table for your organization, the structured next step is a focused assessment. The output is honest enough to support a board-level decision either way: repatriate (with a plan), don’t repatriate (with the reasons), or selective repatriation (with the workload list).

Ænix runs this as a 14- or 28-day Platform Readiness Assessment with the cost-and-cloud-spend workstream emphasized. See the cloud repatriation services page for the engagement details.

Want to dig deeper?

• Cloud repatriation services page — the engagement details and pricing
• Cloud cost optimization — adjacent FinOps trigger
• Data sovereignty — regulatory side of the same shift
• Cozystack — the platform we typically recommend as repatriation destination

The public-sector sovereign-cloud conversation in 2026 is more fragmented than in financial services. There’s no single regulation like DORA driving alignment. Instead, each jurisdiction has its own framework, often layered on top of GDPR, NIS2, and sectoral overlays. A multinational public-sector engagement — or even a national one crossing regions — usually maps against three or more frameworks simultaneously.

The framework landscape

EU level

• EUCS (EU Cybersecurity Certification Scheme for Cloud Services) — proposed EU-wide scheme (adoption still pending). Three assurance levels (Basic, Substantial, High). High level requires substantive sovereignty controls.
• NIS2 — public administration is an Annex I sector; central-government entities are essential entities (Article 3). Article 21 + Article 23 obligations.
• GDPR — personal-data baseline, cross-border-transfer rules under Articles 44-50.

Member-state level

• France: SecNumCloud — strict French national framework. The most demanding EU-member-state sovereign-cloud scheme. Reference standard for several other national initiatives.
• Germany: BSI C5 — German cloud security catalogue. Now widely referenced beyond Germany for DACH operations.
• Italy: ACN — National Cybersecurity Agency frameworks; Polo Strategico Nazionale infrastructure rules.
• Spain: ENS High — Esquema Nacional de Seguridad highest level.

Other member states have variations.

Central Asia and APAC

• Kazakhstan — procurement-mandated sovereignty for public-sector workloads via goszakup.gov.kz / mitwork.kz / zakup.sk.kz.
• Singapore: IM8 — Government IT security standards.
• India: MeitY — Ministry of Electronics IT, including STQC Empanelled CSP framework.
• Australia: IRAP — Information Security Registered Assessors Program; Protected / Secret levels for government workloads.

Sectoral overlays

• Classified workloads (most jurisdictions): national classification overlay
• Healthcare: national health-data sovereignty rules
• Critical-infrastructure: sectoral cybersecurity overlays

What “substantively sovereign” means

A “sovereign cloud” product that doesn’t satisfy all of the following substantively will fail under audit at the High-assurance level of most frameworks:

1. Data residency at every layer

Not just production storage. Backup, observability, CI/CD artefacts, managed-service telemetry, cross-border replication, sub-contractor processing — every layer must respect the residency requirement.

2. Customer-controlled encryption keys

HSM-backed for sensitive data classes. Documented rotation. Emergency- access procedures. Provider personnel cannot extract or copy keys under any circumstances. This is the most common point where hyperscaler “sovereign” offerings fall short — the provider retains operational access to keys, which fails the substantive condition.

3. Open-source platform foundation

For transparency, exit-readiness, and auditability. Closed-source platforms tied to a single vendor’s roadmap fail several frameworks' substantive requirements (even where they pass procurement-policy checks).

4. Supplier-chain transparency

Supply-chain provisions across the frameworks (DORA Art. 28, NIS2 Art. 21(2)(d), national schemes) expect documentation of the supplier chain at least to the second hop. Most hyperscaler-based sovereign-cloud arrangements stop at the first hop (the hyperscaler itself).

5. Air-gap deployment option

For the most sensitive workloads — classified, healthcare-with-strict-residency. Updates flow through controlled channels (customer-side artefact registry, manual approval). Most sovereign-cloud frameworks at High level require air-gap support as an architectural option even if not used by every workload.

6. Audit-trail completeness in standard formats

Logs in standard formats (Syslog, CEF, OpenTelemetry) that the customer’s audit team can consume independently of the platform vendor. Tamper-evident. Retention per the longest applicable regulatory requirement.

7. No phone-home telemetry

Telemetry that leaves the customer perimeter must be opt-in and explicitly documented. Many hyperscaler-managed cloud products have non-optional telemetry channels that fail this criterion.

8. Operational independence under sovereign jurisdiction

Provider personnel access logged and time-limited. Sovereign jurisdiction for the support entity (Ænix has AENIX s.r.o. in Czechia for EU contracts and AENIX INC in Delaware for US contracts). No cross-jurisdictional support routing for sovereignty-sensitive workloads.

What Cozystack-based architecture delivers across frameworks

The architectural pattern that satisfies all major frameworks simultaneously:

• Open-source platform — Cozystack under Apache 2.0, CNCF Project, vendor-neutral substrate. Customer can audit, modify, or replace the platform vendor.
• Customer-held keys — volume encryption is opt-in per storage class and key management is designed with you; Ænix never holds keys.
• Air-gap support — documented air-gapped install workflow for classified-data use cases.
• Self-hosted observability — VictoriaMetrics + VictoriaLogs in jurisdiction; no SaaS-observability residency leak.
• Customer-controlled identity — Keycloak / Active Directory / national IdP integration; Ænix never holds production credentials.
• Multi-tenant Tenant CRD — strong isolation per data class / business unit / sectoral overlay.
• Audit-isolated environments — separate clusters for production, audit, and forensic copy.
• EU jurisdiction support entity — AENIX s.r.o. (Czechia).

The architectural pattern is the same; the certification work is framework-specific. For SecNumCloud-tier engagements, the customer typically engages a certified auditor; Ænix provides the architecture and documentation deliverables, the customer runs the audit cycle.

Procurement realities

Public-sector engagements are procurement-framework-driven in a way that private-sector engagements are not. A few practical realities:

Tender response

Public-sector RFPs typically specify which frameworks must be satisfied (SecNumCloud High, BSI C5, EUCS Substantial, etc.). The response must demonstrate substantive compliance, not just intent. Ænix engagement model includes tender-response support; references can be shared under NDA where the customer allows it.

Multi-year framework agreements

Many public-sector engagements run through framework agreements with specific compliance and exit clauses. Ænix’s commercial entity (AENIX s.r.o. in Czechia for EU; AENIX INC in Delaware for US) is the contracting entity; engagement structure adapts to framework- agreement requirements.

Ænix is not a hyperscaler — that’s the point

Several public-sector mandates explicitly require non-hyperscaler sovereign provision. Ænix’s open-core model — customer hardware, customer-held keys where encryption is enabled, customer operational control, optional Ænix support — fits these mandates structurally rather than via contractual workarounds.

Engagement phases for public-sector

Phase 0 — Framework scoping

Confirm applicable frameworks. Identify the highest-bar one (usually SecNumCloud High for French, BSI C5 for German, EUCS High where EU-wide requirements apply, national procurement rules elsewhere). Design the architecture against the highest bar; map down to the others.

Phase 1 — Architecture and procurement-response work

Produce procurement-response artefacts: technical proposal, framework compliance mapping, reference architecture, sample evidence catalogue. Typical duration: 2-4 months.

Phase 2 — Phase-1 platform build (per Public Cloud Platform /

Private Cloud Platform models)

Multi-DC deployment, air-gap option enabled if applicable, sovereign identity integration, audit-isolated environments. A national multi-region programme runs a 3-6 month pilot, then 9-18 months to full multi-region operation; a single-agency private cloud is a 3-12 month build after a 14- or 28-day assessment.

Phase 3 — Certification cycle

Customer engages accredited auditor; Ænix provides architecture documentation, control mapping, evidence catalogue. Ænix engineers participate in technical interviews with the auditor where allowed. Typical certification cycle: 6-12 months parallel to Phase 2.

Phase 4 — Production operations

Customer team operates the platform with Ænix advisory and a Plus or Enterprise support tier (see /pricing/). Annual recertification cycle (most frameworks).

Certified production comes later than in private-sector engagements because of the certification cycle, but the certification value compounds — once certified, the platform retains certification with annual recertification rather than per-engagement.

Ænix’s existing public-sector posture

Ænix contracts through AENIX s.r.o. (EU) and AENIX INC (US) and works within public-sector procurement frameworks. Specific engagements are confidential; references are available under NDA in the discovery call.

When this engagement model fits

Strong fit:

• National sovereign cloud initiatives (public, public-private partnership, sovereign cloud operator)
• EU member-state regional / sectoral cloud programmes
• Classified-data hosting with an air-gap requirement
• Healthcare sovereign cloud at national or regional level
• Education / research consortia with multi-decade planning horizon

Marginal fit:

• Single ministry / agency procurement with smaller scope — may fit Private Cloud Platform rather than Public Cloud Platform

Poor fit:

• Workloads where hyperscaler-managed cloud is already framework- compliant (some specific procurement frameworks)
• Organisations without sovereignty pressure (use the private-sector product matching the workload profile)

Where to dig deeper

• Public-sector industry page — the commercial landing
• Data sovereignty services — buyer-trigger sovereignty landing
• NIS2 compliance services — for the NIS2 overlay (public administration is Annex I)
• Sovereign Cloud Builder services — the engagement type
• Build sovereign cloud — playbook for EU and Central Asia — EU and Central Asia sovereign cloud playbook
• Data residency requirements in 2026 — per-layer residency walkthrough

Most hosting providers run Ænix Public Cloud Platform at provider scale: the productized installer puts the platform live in weeks once hardware is ready, priced from the published support tiers. This post is about the other end — the operator-scale programme. The question there is not “should we use Cozystack?” — that’s already decided. It’s “we are launching a cloud product at national or tier-1-customer scale; what does the partnership with Ænix look like across a 3-6 month pilot and the 9-18 months to full multi-region operation?”

Who runs Public Cloud Platform at operator scale

Five buyer profiles dominate operator-scale Public Cloud Platform engagements:

1. Tier-1 telcos / national operators — incumbent telecom operators launching or scaling a public cloud as part of their product portfolio. Often paired with sovereignty positioning (“our sovereign cloud”, “national cloud”).
2. Big banks operating their own cloud — the bank consumes its own cloud product for internal workloads and, sometimes, sells capacity to its customer base.
3. Sovereign cloud initiatives — government-mandated cloud products, sometimes with public-private-partnership structure, with explicit sovereignty requirements and regulator alignment.
4. Hosting providers at large scale — providers above ~5,000 customers where the Public Cloud Platform operational model needs scaling into multi-region with multi-DC active/active.
5. National AI/GPU operators — sustained inference + training capacity for sectoral customers (banks, healthcare, public sector) where AI sovereignty is a national-level requirement.

All five share the same operational reality: multi-region or multi-DC active/active; multi-million-euro infrastructure investment; customer-facing SLAs that map to national regulator expectations; and a partnership model with Ænix that lasts years, not months.

What an operator-scale build adds

Multi-region / multi-DC active/active

Single-DC deployments are served by Public Cloud Platform at provider scale (or Private Cloud Platform for internal use). Operator-scale builds assume from day one that the customer needs active/active across regions or datacentres with cross-DC replication tuned for RTO/RPO targets. The platform’s control plane, observability, identity, and storage layers all design for multi-region from the foundation rather than retrofitting.

Service-catalog depth

A provider-scale build exposes ~20 managed services. An operator-scale build typically targets 30-50+ services across compute, storage, networking, managed databases, observability, AI/GPU, message queues, search, content delivery, security tooling. Cozystack’s package architecture (Package + PackageSource + ApplicationDefinition resources, as of v1.x) supports the catalog expansion.

Operations team at scale

10-30+ engineers running the platform, depending on customer count and SLA. An operator-scale engagement includes operations team hiring and training as a substantial workstream — not “you find people, we’ll train them” but “we design the org structure with you, participate in interviews, do hands-on training, and provide escalation support (Plus or Enterprise tier) for the first 12-18 months while your team builds confidence.”

Regulator and sovereignty alignment

Whatever the sovereignty framework is in the customer’s market — SecNumCloud, BSI C5, EUCS, sectoral overlays, national procurement mandates — the architecture is designed to satisfy it substantively, not just contractually. Compliance evidence catalogue is a deliverable.

Customer-facing brand engineering

Beyond Cozystack Dashboard customisation, an operator-scale engagement includes brand-engineering work: customer portal that looks like a top-tier cloud product, not a customised Cozystack instance. UX flows tuned to how customer’s customers think about ordering, configuring, paying. Designer-led, not engineering-led.

How an operator-scale engagement phases

Phases 0 and 1 form the pilot and take 3-6 months together. Phases 2-4 take 9-18 months to full multi-region operation, overlapping where the teams allow.

Phase 0 — Discovery and partnership formation (start of the pilot)

Before engineering, agreement on:

• Strategic objectives (what cloud product, what customer base, what competitive positioning)
• Regulatory scope (which frameworks bind the platform)
• Org structure (who owns what; how Ænix and customer teams interact)
• Commercial structure (engagement model, IP, support model post-go-live)
• Roadmap (phasing of services, geographic expansion, SLA tiers)

Output: signed engagement plan with named workstream leads on both sides.

Phase 1 — Foundation (rest of the pilot)

Hardware procurement and racking. Talos / Cozystack platform deployment in the first datacentre. Storage layer (LINSTOR/DRBD at scale). Networking foundation. Identity integration with customer’s existing workforce identity (Keycloak / Okta / Active Directory / sovereign IdP). Initial observability stack.

End state: working platform, single region, internal access only. Not yet customer-ready.

Phase 2 — Multi-region foundation

Second datacentre stood up. Cross-DC replication validated. Federated identity. Multi-region storage replication (LINSTOR async or Ceph cross-region). Disaster recovery patterns tested. Compliance documentation foundation built.

End state: multi-DC platform, internal access, RTO/RPO validated against targets.

Phase 3 — Service catalog buildout

Service-by-service rollout. Start with foundational services (compute, storage, basic networking, managed PostgreSQL). Layer in managed service families (databases, queues, caches, search, observability). Add product-specific services (GPU, AI inference, sectoral compliance tooling).

Each service goes through: deployment → internal testing → friendly- customer pilot → production GA. Cohort-based rollout, not big-bang.

Phase 4 — Customer onboarding and limited GA

Customer-facing portal launched (brand-engineered). Billing integration validated end-to-end. Support runbooks documented. First 10-50 friendly customers onboarded. SLA monitoring operationalised.

End state: cloud product live with first customer cohort, billing and support workflows proven.

Phase 5 — General availability and scale (ongoing)

Open market launch. Marketing and sales activated. Operations team scales to support customer growth. Ænix escalation support (Plus or Enterprise tier) continues until the customer team is ready to absorb it (typically 12-24 months post-GA).

Subsequent phases are roadmap-driven: new services, new regions, new sectoral SKUs.

Where multi-million-euro cloud projects fail

Three failure patterns we’ve seen across the industry:

1. Under-investing in brand engineering

Engineering-led platform with engineering-grade UX. Customers click around, find it functional but unappealing, sign up for hyperscaler instead. An operator-scale engagement includes design partnership explicitly to avoid this.

2. Operations team sized for go-live, not 18-month-out volume

Cloud products grow exponentially during the first year of GA if positioning is right. Operations teams sized for go-live customer count get overwhelmed at month 6-12. Plan operations capacity for 18-month-out volume; hire ahead.

3. Regulator dialog deferred

Sovereignty positioning depends on regulator endorsement (explicit or implicit). Projects that defer the regulator conversation until late-phase find themselves rebuilding architecture to satisfy expectations they could have designed for at the start. Engage regulators in Phase 0-1, not Phase 4.

When an operator-scale programme is the right answer

Strong fit:

• Tier-1 telco / national operator / large bank / sovereign cloud initiative
• Multi-region or multi-DC operational reality
• 5,000+ target customer count or strategic customer base
• Multi-million-euro budget envelope across a multi-year programme
• Sovereignty / regulator positioning is core to value proposition
• Senior executive sponsorship (CIO or CTO level minimum)

Marginal fit:

• Large hosting providers below tier-1 telco scale — Public Cloud Platform at provider scale, extended region by region, often fits better than a full operator programme, depending on growth profile
• AI/GPU-focused operators where the AI workload dominates — the Ænix AI Platform may fit better, with selective Public Cloud Platform components

Poor fit:

• Smaller hosting providers — Public Cloud Platform at provider scale (from the published price list) fits substantially better on economics and operational model
• Regulated enterprises consuming cloud rather than producing it — Private Cloud Platform is the right answer

Engagement structure

• Discovery call (executive level, 60-90 min) — strategic fit assessment
• Platform Readiness Assessment (fixed price, 28 days full) — input to partnership formation; output is a signed engagement plan
• Pilot (Phases 0-1, 3-6 months), then Phases 2-4 (9-18 months to full multi-region operation)
• Support subscription (ongoing) — Plus or Enterprise tier (see /pricing/) until the customer team is ready to absorb escalation

Engagement size: multi-year programme, quoted per RFP.

Where to dig deeper

• Public Cloud Platform landing — feature list, product-specific FAQ
• Public Cloud Builder services — engagement details
• Sovereign Cloud Builder services — for the sovereignty-specific variant
• Build sovereign cloud — playbook for EU and Central Asia — sovereign cloud architectural patterns

The post-Broadcom virtualization market has three main options this article compares: Proxmox VE and Cozystack (both open source) and VMware itself (XCP-ng is a less common fourth). Each has a different architectural target. Picking the right one is mostly a function of scale and use case.

Proxmox VE — SMB-friendly, VM-focused

Architecture: KVM + LXC + ZFS + Ceph (community), single-cluster Proxmox VE, no native multi-tenancy.

Strengths:

• Mature, stable, easy to install.
• Strong community.
• AGPLv3 licence, commercial subscription available.
• Excellent for single-team or single-tenant deployments.
• Proxmox Backup Server is good.

Limits:

• Multi-tenancy through namespaces and permissions; not designed for hard isolation.
• Service catalog beyond VMs (managed databases, S3, etc.) requires manual integration.
• Service-provider use cases (billing per tenant, self-service portal) require external software.
• Federation across clusters is heavier than Kubernetes.

Best for: SMB IT departments, labs, dev environments, single-tenant private virtualization, teams under ~50 hosts.

VMware (post-Broadcom) — enterprise legacy

Architecture: vSphere + vSAN + NSX + vCloud Director + vRealize/Aria. Closed source, subscription-licensed.

Strengths:

• Mature, well-known, extensive ecosystem.
• Strong enterprise tooling integration.
• Broad operational expertise in market.

Limits:

• Subscription-only licensing (post-Broadcom 2023).
• 2-5× price increases on renewal observed across our pipeline.
• Vendor lock-in across the stack.
• Sovereignty concerns (US-headquartered vendor).

Best for: Existing VMware estates that haven’t yet been triggered out by economics. New deployments rarely choose VMware in 2026.

(See VMware alternative for migration guidance.)

Cozystack — open-source, Kubernetes-native, multi-tenant

Architecture: KubeVirt + Cilium + Kube-OVN + LINSTOR (DRBD) + Tenant CRD + Cozystack Dashboard. Open-source CNCF Project.

Strengths:

• Kubernetes-native virtualization — same platform for VMs, containers, databases.
• Multi-tenancy structural (Tenant CRD) — production-grade for service providers and regulated multi-tenant.
• First-class managed database, S3 object storage, GPU services.
• Apache 2.0 licence, no per-CPU pricing.
• Air-gapped deployment supported.

Limits:

• Newer than Proxmox or VMware; smaller community.
• Kubernetes operational expertise required (mitigated by Ænix support tier).
• Not optimized for single-tenant SMB use case (Proxmox better here).

Best for: Service providers, regulated enterprises, multi-team platforms, AI/GPU operators, sovereign-cloud builders.

Comparison matrix

Migration paths

Proxmox → Cozystack

VM images (qcow2) import directly into KubeVirt CDI. Multi-tenant model designed during migration (Proxmox didn’t have one to migrate). Storage and network re-architecture. Typical: a 14- or 28-day assessment + 3-9 months implementation.

VMware → Cozystack

KubeVirt-based migration with image conversion. Windows VMs supported; specific tooling for VMware Tools cleanup. Multi-tenant model maps from vCloud Director to Tenant CRD. (Full guidance: VMware alternative landing.)

Proxmox → VMware

Rare in 2026; reverse migration usually doesn’t make economic sense post-Broadcom.

How to choose

1. You’re under 50 hosts, single-tenant, mostly VMs: Proxmox VE.
2. You’re a service provider, multi-tenant cloud, regulated multi-tenant: Cozystack.
3. You’re already on VMware and the budget supports staying: stay (but plan an exit). If renewal pressures bite: see VMware alternative.
4. AI/GPU workloads at scale: Cozystack (KubeVirt + GPU operators).
5. Pure container workloads, no VMs: vanilla Kubernetes (Cozystack still works but is over-spec).

Proxmox VE is one of the most successful open-source virtualisation platforms of the last decade. Mature, easy to install, strong community, AGPLv3 with commercial subscription. We talk to a lot of operators who started on Proxmox, grew, and are evaluating what comes next.

Crucially: Proxmox is the right answer for many of them. This article covers when migration is warranted and when it’s premature.

Where Proxmox keeps winning

Proxmox VE remains the right answer for:

• SMB IT departments — small-to-mid businesses running 10-50 virtualised workloads on premises, no customer-facing multi- tenancy needs
• Single-tenant labs and dev environments — Proxmox’s operational simplicity beats any heavier alternative
• Mature operators with stable customer base under ~200 customers — Proxmox’s commercial economics still work; migration cost would exceed the value
• Mostly-VM workloads — Proxmox’s KVM + LXC scope fits cleanly
• Existing operators with deep Proxmox expertise and stable team — switching cost includes team retraining

If your situation matches these, don’t migrate. The Ænix Public Cloud Platform is over-engineered for SMB single-tenant operation. We say this in discovery calls rather than push the engagement.

When Proxmox is being outgrown

Migration warrants serious evaluation when at least three of these hold:

1. Customer count growing past ~300

Proxmox’s multi-tenancy model (pools, realms and permissions, not hard isolation) starts to feel thin above ~300 customer-facing tenants. Per-customer audit trails, isolation guarantees, and quota enforcement become operational pain.

2. Customers asking for services beyond VMs

Managed PostgreSQL, MariaDB, MongoDB, Redis, Valkey, Kafka, S3-compatible object storage, tenant Kubernetes clusters, GPU services. Proxmox’s scope is VMs + LXC; everything else is bolted on with manual integration or external systems.

3. WHMCS or similar customer-management integration

Proxmox has WHMCS integration, but the service catalog beyond VMs is manual integration work. Ænix Public Cloud Platform adds a WHMCS integration (a proprietary Ænix module, not part of open-source Cozystack) that covers the full service catalog.

4. Multi-DC active/active

Proxmox clustering is single-DC. Geographic distribution requires manual cross-cluster replication patterns. Cozystack handles multi- DC active/active as a first-class deployment mode.

5. Container-native customer demand

Customers want tenant Kubernetes clusters or container-native service catalogs. Proxmox can host containers via LXC but isn’t the right operational model for tenant-facing Kubernetes-as-a-service.

6. Recurring licence / subscription pressure on commercial Proxmox

Proxmox’s commercial subscription is competitive but real cost. Operators with growing infrastructure footprint sometimes find the total subscription cost approaching what Ænix charges for Public Cloud Platform support — at which point the service-catalog and operational upside of Cozystack tips the decision.

Architectural mapping: Proxmox → Cozystack

Two areas need redesign rather than 1:1 mapping:

• LXC vs Kubernetes containers — Proxmox LXC is system-container (full OS image), Kubernetes container is application-container (single process or small set). Workloads using LXC for system- container patterns either migrate to KubeVirt VMs or get refactored.
• Multi-tenancy model — Proxmox tenant model (pools, realms and permissions) versus Cozystack Tenant CRD (Kubernetes-native). Customer-facing isolation is stronger in Cozystack; operational abstraction is different.

Migration phases

Phase 0 — Assessment (14 or 28 days)

Inventory: customer count, customer-facing services consumed, VM count, OS mix, LXC usage, storage tiers, network topology, backup patterns, WHMCS / customer-management integration.

Honest TCO comparison: current Proxmox + commercial subscription + operational team versus Ænix Public Cloud Platform + hardware refresh + Ænix support tier. For operators under ~300 customers, this often shows Proxmox staying competitive; above ~500, Cozystack typically wins on service-catalog and operational depth.

Output: go/no-go decision with quantified justification.

Phase 1 — Cozystack foundation (live in weeks)

The platform goes live in weeks once hardware is ready, using the productized installer; catalogue and brand work continue alongside the pilot. Cozystack platform deployed on new hardware or repurposed Proxmox hardware (commodity x86 servers move easily). Cilium networking configured. LINSTOR storage operationalised. Identity integration (typically Keycloak + customer IdP). Cozystack Dashboard brand customisation matching the operator’s existing brand.

WHMCS integration validated end-to-end. Service catalog populated with the operator’s chosen services (VMs first, managed databases next, S3 then, expanding from there).

Phase 2 — Pilot customer migration

5-20 friendly customers migrated to Cozystack as the first cohort. Pattern per customer:

1. Customer VMs converted from Proxmox qcow2 to KubeVirt-compatible format
2. Network configuration translated (Proxmox bridges → Cilium ClusterPool + NetworkPolicies)
3. Storage migrated (ZFS / Ceph volumes → LINSTOR in Cozystack)
4. Customer-side validation window (7-14 days)
5. DNS / load balancer cutover

During the pilot, customer support team builds operational familiarity with Cozystack. Documentation patterns shake out.

Phase 3 — Production migration cohorts

Cohorts of 30-100 customers at a time. Same per-customer pattern as pilot, with operational efficiency improvements as the team internalises the workflow.

LXC-using customers receive special handling: either system-style KubeVirt VM (1:1 replacement) or refactor to Kubernetes-native application container (depending on customer’s preference and support).

Phase 4 — Proxmox decommission

As migration cohorts complete, Proxmox hardware moves into the Cozystack cluster. Proxmox subscription wound down per renewal cycle. Proxmox Backup Server data archived per customer agreements.

Timeline realities

For typical mid-size hosting provider (300-1,000 customers):

• Phase 0: 14 or 28 days
• Phase 1: platform live in weeks once hardware is ready
• Phases 2-3 (pilot and production cohorts): typically 3-9 months
• Phase 4: alongside the last cohorts, timed to Proxmox renewals

Total: the platform is live in weeks once hardware is ready; moving workloads and customers typically takes 3-9 months

For larger operators (1,000-5,000 customers), Phase 3 runs longer for sustainable cohort pacing; the assessment sets the schedule.

Where Proxmox-to-Cozystack migrations stumble

1. LXC workloads

If a substantial fraction of customer workloads use LXC for system-container patterns (e.g., per-customer LAMP stack as a single LXC), the migration to Kubernetes-native containers requires refactoring. The alternative is running them as KubeVirt VMs (1:1 mapping but heavier resource footprint). Plan time for this in Phase 0.

2. Customer-facing API divergence

Some customers built tooling against the Proxmox API. Cozystack exposes Kubernetes API + Cozystack Dashboard API; the contracts differ. Customer-facing migration support (documentation, sometimes API compatibility shim) is engagement work.

3. Operations team training

Proxmox operators are comfortable with the Proxmox web UI and the imperative qm / pct / pvesh CLI tools. Cozystack expects GitOps for production changes. Operations team needs 4-8 weeks of focused training plus 3-6 months of practice. Ænix engagement includes training; customer investment in the transition is also required.

4. ZFS-specific workloads

Some customers chose Proxmox specifically for ZFS-on-host features (advanced snapshots, ZFS-replicated backups). Cozystack ships LINSTOR (DRBD); ZFS-specific operational patterns don’t translate. Customer dialogue about feature equivalence is part of Phase 0.

Versus other alternatives

Versus building it yourself on raw KVM + libvirt + Kubernetes: Same trade-offs as for any open-source-build option. Cozystack gets a multi-tenant platform to production in weeks to a few months; raw builds take 12-24 months to reach the same level. For operators with strong platform engineering capacity, the raw-build is a credible alternative.

Versus VMware (post-Broadcom): Proxmox-to-VMware migration is rare in 2026 — reverse migration usually doesn’t make economic sense post-Broadcom.

Versus Nutanix: Nutanix AHV is closed-source proprietary KVM. For operators valuing open-source substrate, Cozystack wins on that property alone. For operators valuing integrated commercial support without open-source overhead, Nutanix wins.

Versus OpenShift Virtualization: Both are KubeVirt-based. OpenShift fits existing Red Hat / OpenShift customers; Cozystack fits operators preferring open-source-first procurement and lighter operational footprint.

When this engagement model fits

Strong fit:

• Hosting provider or MSP with 300+ customers
• Growth trajectory toward 1,000+ customers
• Customer demand for services beyond VMs
• Multi-DC operational reality
• Budget for a migration programme (typically 3-9 months of customer moves)

Marginal fit:

• 200-300 customer providers — borderline; depends on growth trajectory and service-catalog ambition

Poor fit:

• SMB IT (<100 internal VMs) — Proxmox is still better
• Lab / dev environments — Proxmox simplicity wins
• Sub-200-customer hosting providers — poor fit for a full migration programme; consider a greenfield service line on Ænix Public Cloud Platform at provider scale instead

Engagement structure

• Discovery call (30 min, free)
• Platform Readiness Assessment (fixed price, 14 days focused or 28 days full) — go/no-go with TCO comparison
• Pilot deployment — Cozystack stood up (live in weeks once hardware is ready), 5-20 friendly customers migrated
• Cohort migration — customer migration in cohorts; pilot and cohorts together typically take 3-9 months
• Proxmox decommission (parallel) — as cohorts complete
• Support subscription (ongoing) — Plus or Enterprise support tier for 24×7 coverage (see /pricing/)

Where to dig deeper

• Proxmox migration hub — commercial landing
• Proxmox vs VMware vs Cozystack comparison — decision matrix
• Proxmox alternative — alternative-focused commercial landing
• Hosting providers industry page — industry-specific positioning
• Public Cloud Platform economics for hosting providers — unit-economics walkthrough
• Hosting provider platform modernization — modernisation pattern

The decision to deploy a private LLM is increasingly easy to make and surprisingly hard to execute well. The decision is easy because the trigger is usually clear: regulated data, sectoral rules, or the economics of inference at scale make a hyperscaler model API the wrong answer. Execution is hard because the supporting infrastructure — GPU scheduling, model serving, observability, multi-tenancy, audit-readiness — is more work than the LLM itself.

When a private LLM is the right answer

Three trigger profiles dominate the conversations that lead to private LLM deployment.

Trigger 1 — regulated data class

The data your AI must process is bound to a jurisdiction by regulator, sector, or contract. Examples: customer financial records under DORA / sectoral banking rules; patient-data in healthcare; classified or sensitive public-sector data; legal-privilege content; internal IP whose exposure to a third-party model would constitute a contractual or competitive breach.

For these workloads, sending data to a hyperscaler model API — even one with a “private” deployment branding — fails the substantive requirement. The data must stay in the customer’s perimeter, and so must the model.

Trigger 2 — inference economics at scale

For 24/7 inference workloads at sustained throughput, hyperscaler GPU economics break down. A workload that consumes a few hundred GPU-hours per day at peak burst rates can cost more on a hyperscaler than a dedicated cluster does in a year. The crossover point varies by GPU class and utilization, but moves a lot of inference workloads onto on-premise economics around 30-50% sustained utilization.

Trigger 3 — auditability and reproducibility

Regulator dialog increasingly requires “exactly which model produced this output, with which weights, with which input data, on which hardware, at what time.” Hyperscaler model APIs answer the first question (model name + version) but not the others. Private deployment makes all of them auditable.

If your situation matches one of these triggers strongly, or two weakly, private LLM deployment moves from “interesting option” to “obvious next step.”

What goes into a real private LLM stack

A production private LLM stack has six layers, all of which need an answer:

1. Hardware — GPUs, CPUs, networking, storage
2. Platform — Kubernetes, virtualization, GPU scheduling
3. Serving stack — model server, batching, autoscaling
4. Model layer — open-weight models, fine-tuned variants, embeddings
5. Application layer — RAG, agent frameworks, LLM gateway
6. Operations — observability, audit, cost management, on-call

Skipping any layer — particularly platform and operations — produces a PoC that doesn’t survive contact with production.

Layer 1: hardware

The right GPU depends on workload, model size, and budget. The current production-ready landscape:

• NVIDIA H100 / H200 — the workhorse for fine-tuning and high-throughput inference of mid-to-large models. Expensive, available with lead time, broadly supported.
• NVIDIA Blackwell (B100/B200) — newer, higher memory bandwidth, well-suited for the largest models. Lead times longer.
• NVIDIA L40S — 48 GB GPU memory; good fit for inference of smaller models or as part of a multi-tenant inference fleet.
• NVIDIA A100 — older but still cost-effective for many inference workloads; second-hand market reasonable.
• AMD MI300 / MI325 — credible alternative for some workloads; ROCm tooling still maturing relative to CUDA.
• Specialized accelerators (Groq, Cerebras, etc.) — strong for specific use cases; ecosystem narrower.

For most regulated-industry deployments, a fleet of H100/H200 (or A100 for cost-sensitive) plus L40S for tenant-fleet inference is the typical answer. NVIDIA data-centre GPUs are supported through the NVIDIA GPU Operator (passthrough to VMs, sharing via HAMi).

The CPU and storage sizing are also important — but largely follow standard practice once GPU sizing is set. Networking matters significantly for multi-GPU training (NVLink, InfiniBand, RoCE) and somewhat less for inference (where 25-100 Gbps Ethernet is usually sufficient).

Layer 2: platform

The platform sits between hardware and applications. For private LLM, the right platform answers:

• How GPUs are scheduled across workloads — full passthrough, MIG (NVIDIA Multi-Instance GPU), time-slicing, or virtual GPU (vGPU)
• How VMs and containers coexist (most data-science workloads run as containers; some legacy or notebook-heavy workloads as VMs)
• How multi-tenancy is enforced — tenant isolation, quota, scoped audit
• How observability, identity, and storage integrate

Kubernetes-native virtualization platforms (Cozystack, OpenShift Virtualization, vendor-led variants) are increasingly the default because they answer all of these in one stack.

Cozystack supports:

• Container-based AI workloads with Kubernetes GPU scheduling: whole-GPU allocation through the NVIDIA GPU Operator, fractional sharing (GPU memory and compute cores) through HAMi
• VM-based AI workloads through KubeVirt with passthrough of whole GPUs or NVIDIA vGPU (requires your NVIDIA vGPU licence)
• Multi-tenant isolation through Tenant CRD with per-tenant GPU quotas
• MIG and time-slicing: on the roadmap, not shipped today

OpenStack-based or VMware-based legacy platforms can be retrofitted to host AI workloads, but typically with more operational friction and less native Kubernetes integration.

Layer 3: serving stack

The serving stack runs the model. Options:

• vLLM — high-throughput serving for transformer models with PagedAttention. Default choice for most inference workloads.
• NVIDIA Triton Inference Server — broader model-format support; strong for mixed workloads (LLM + vision + embedding + classical ML).
• TGI (Text Generation Inference) — Hugging Face’s serving stack; some specific feature niches.
• llama.cpp / Ollama — for smaller models, single-machine deployments, or development/PoC.
• Custom — for specialized workloads that need direct framework access.

For multi-tenant production: vLLM (or Triton) with autoscaling, batched inference, and a request router (LiteLLM, Portkey, or custom). The choice between vLLM and Triton is mostly a function of the model formats and existing ML pipelines.

Layer 4: model layer

Open-weight models in production-ready 2026 landscape:

• Llama (Meta) — large family covering 1B to 405B+, licence permits commercial use with caveats
• Mistral — Mixtral (MoE) and Mistral Large; commercial licence for some, Apache 2.0 for older
• Qwen (Alibaba) — strong multilingual, including non-English; Apache 2.0
• DeepSeek — strong reasoning models; licence varies
• Phi (Microsoft) — small models with surprising capability; MIT
• Gemma (Google) — small/medium-size; licence permits commercial use

Selection depends on:

• Language requirement (multilingual vs English-only)
• Workload type (chat / RAG / code / vision / embedding)
• Cost-per-token target at expected throughput
• Licence terms (commercial use, attribution, redistribution)

For most regulated-industry deployments: a primary model in the 7B-70B range for general use, plus smaller models (Phi, Gemma) for cost-sensitive paths, plus an embedding model for RAG. Fine-tuning happens for domain-specific accuracy needs.

Layer 5: application layer

Most production deployments add:

• LLM gateway — request routing, rate limiting, audit logging, cost tracking. Examples: LiteLLM, Portkey, custom.
• RAG infrastructure — vector database (Weaviate, Qdrant, Milvus, pgvector), document indexing pipeline, retrieval orchestration.
• Agent framework (where applicable) — LangChain, LlamaIndex, custom.
• Observability for LLM — token usage, latency p50/p95/p99, error rates, content filters.

The application layer is where model-provider switching costs concentrate. A well-engineered LLM gateway makes the underlying model swappable; a poorly-engineered one locks the application to a specific model API.

Layer 6: operations

The post-deployment steady state is where most private LLM projects underperform expectations. The operational surface includes:

• GPU utilization monitoring — under-utilized GPUs are wasted budget; over-utilized GPUs cause queueing
• Model lifecycle management — when do you upgrade Llama 3.x to 4.x? When do you re-fine-tune? Who owns the regression-test suite?
• Cost tracking by tenant / team / workload — without per-tenant tracking, cost allocation becomes unmaintainable
• Audit trail — for regulator dialog, every inference request must trace to: model + version, weights, input, output, requesting user, timestamp
• On-call — GPU failures, OOM events, queue backups
• Capacity planning — when do you buy more GPUs? When do you retire older ones?

A team that under-staffs the operations layer ends up with a stack that “works” but isn’t production-grade for regulator audit or business-continuity standards.

Common deployment patterns

Pattern 1: single-tenant inference cluster

Smallest reasonable deployment. Single inference workload, single team, single model family. Typical hardware: 4-16 H100/H200 or 8-32 L40S. Platform: bare Kubernetes or Cozystack. Serving: vLLM. Best for: regulated workloads with one application owner; PoCs.

Pattern 2: multi-tenant inference fleet

Multi-team or multi-application deployment. Multiple model families, per-tenant quotas, shared observability. Typical hardware: 16-128 GPUs across multiple model classes. Platform: Cozystack with Tenant CRD or OpenShift with namespaces + quotas. Serving: vLLM or Triton with autoscaling and a request router. Best for: enterprise platform teams supporting multiple data-science teams.

Pattern 3: inference + fine-tuning + RAG

Full AI platform. Inference fleet, dedicated fine-tuning capacity, RAG infrastructure with vector database, embedding model, LLM gateway. Typical hardware: 32-512 GPUs across roles. Platform: Cozystack or full Kubernetes with operators. Best for: financial services, healthcare, public sector with sustained AI program.

Pattern 4: air-gapped sovereign deployment

Air-gapped or restricted-egress deployment for the most sensitive workloads. No internet egress; updates through controlled channels (Harbor mirror, internal package registry, manually distributed model weights). Typical hardware: customer-supplied. Platform: Cozystack with air-gap install. Best for: classified, defence-adjacent, healthcare-with-strict-residency.

Common pitfalls

Pitfall 1: skipping the platform layer

Teams deploy vLLM on a couple of bare-metal boxes, call it a private LLM deployment. Works for a PoC. Falls over the first time GPU demand exceeds available capacity, or the first time multi-tenant isolation matters, or the first time the regulator asks for an audit trail.

Pitfall 2: model-API-as-private-LLM

Some vendors market a SaaS endpoint with privacy controls as “private LLM.” The data still leaves the customer’s perimeter, even if the privacy clause is strong. For trigger-1 (regulated data) workloads, this fails the substantive requirement.

Pitfall 3: under-sizing memory

The most common mistake in initial sizing: not accounting for KV cache memory that scales with context length and batch size. A model that “fits” by parameter count may not fit at the operational batch size. Right-sizing requires actual benchmark with realistic context lengths.

Pitfall 4: ignoring the cost-allocation problem

Without per-tenant cost tracking, the platform’s economics become opaque to finance. Teams over-request GPU; finance pushes back; the platform team has to retrofit cost tracking after the fact. Build it from day one.

When NOT to deploy private LLM

Private LLM is the wrong answer when:

• Your AI workload is small (under ~10K requests/day) and not regulated. The economics of running dedicated GPU fleet don’t beat hyperscaler on this scale.
• Your team has no Kubernetes / platform-engineering capacity. A private LLM platform is an operational commitment.
• Your model needs are at the absolute frontier (current GPT-class capability with all bells and whistles). The largest open-weight models close the gap rapidly, but the very latest frontier capability still trails.
• Your data is not actually regulated, your spend is not actually growing, and the trigger is more “we want our own thing” than a substantive driver. The work is real and substantial; the trigger has to be real too.

A good engagement is honest about these cases. The Ænix engagement specifically does not push private LLM when the alternative is the right answer.

Want to dig deeper?

• Sovereign AI services page — engagement details
• Data sovereignty — regulator-side trigger
• DORA compliance — financial-services trigger
• Cozystack — the platform we run AI workloads on

The private cloud landscape has shifted significantly in the last 3 years. Broadcom-induced VMware migrations, sovereignty mandates, AI workload economics, and FinOps pressure have all reshaped what “private cloud” means and what providers serve it.

Two distinct things called “private cloud”

The terminology is overloaded. “Private cloud” means either:

• Private cloud platform — software you deploy on infrastructure you control. Examples: VMware VCF, Cozystack, OpenStack, OpenShift Virtualization, Proxmox VE.
• Private cloud provider — a vendor that provides dedicated infrastructure (single-tenant) which you consume. Examples: IBM Cloud Private, Oracle dedicated regions, hyperscaler “sovereign” regions, regional cloud providers (OVHcloud, Hetzner, or a regional provider running Ænix Public Cloud Platform).

Both are valid; they answer different questions. This article focuses primarily on platforms (the software layer); providers come up where relevant.

Open-source platforms

Cozystack

Licence: Apache 2.0, CNCF Project. Architecture: Kubernetes-native virtualization (KubeVirt) + Cilium and Kube-OVN networking + LINSTOR (DRBD) block storage + SeaweedFS object storage + Tenant CRD multi-tenancy + Cozystack Dashboard self-service. Maintainer: Ænix (open-source, community-governed). Best for: Service providers, sovereign-cloud builders, regulated multi-tenant, AI/GPU operators with sustained workloads. Strengths: Single platform for VMs + containers + databases + S3 + GPU. Multi-tenancy structural. Light operational footprint relative to OpenStack. Open-source, no vendor lock-in. Limits: Newer than OpenStack; smaller community.

OpenStack

Licence: Apache 2.0, OpenInfra Foundation. Architecture: Nova compute + Neutron network + Cinder block + Swift object + Keystone identity + Horizon UI + many other components. Maintainer: OpenInfra Foundation; commercial distros from Red Hat, Canonical, Mirantis. Best for: Large telecom operators, government clouds, OpenStack-fluent teams. Strengths: Mature, broad community, many vendor options. Limits: Operationally complex; harder to find OpenStack engineers in 2026; less Kubernetes-native.

OpenShift Virtualization (Red Hat)

Licence: Red Hat commercial subscription. Architecture: OpenShift Kubernetes + KubeVirt + Red Hat ecosystem. Maintainer: Red Hat / IBM. Best for: Existing Red Hat customers, enterprises with Red Hat procurement. Strengths: Strong commercial support, mature. Limits: Subscription pricing; tied to Red Hat / IBM relationship.

Proxmox VE

Licence: AGPLv3 + commercial subscription. Architecture: KVM + LXC + ZFS + Ceph (community). Maintainer: Proxmox Server Solutions GmbH. Best for: SMB virtualization, single-tenant, labs. Strengths: Mature, easy to install, strong community. Limits: Limited multi-tenancy; service catalog beyond VMs requires manual integration.

Apache CloudStack

Licence: Apache 2.0. Architecture: Hypervisor-agnostic (XenServer / KVM / VMware), service-provider-oriented. Best for: Service providers in markets where CloudStack remains established (some EU, MENA, APAC). Strengths: Service-provider features mature; multi-tenancy native. Limits: Smaller community than alternatives; less Kubernetes-native.

Commercial / closed-source platforms

VMware (VMware Cloud Foundation)

Licence: Subscription-only post-Broadcom. Architecture: vSphere + vSAN + NSX + vCD + vRealize. Best for: Existing VMware estates that haven’t yet been triggered out by economics. Strengths: Mature, well-known, extensive ecosystem. Limits: Subscription pricing increases (2-5× observed); vendor lock-in; sovereignty concerns.

Nutanix

Licence: Subscription, multiple tiers. Architecture: AHV (proprietary KVM-based) + Files + Volumes + Era (databases). Best for: Existing Nutanix HCI customers, enterprises preferring appliance model. Strengths: Operationally simple, integrated stack. Limits: Closed source; appliance lock-in; less flexible than open alternatives.

Scale Computing HC3

Licence: Subscription. Architecture: KVM-based hyperconverged appliance. Best for: ROBO / edge / SMB. Strengths: Operationally simple. Limits: Smaller scale ceiling; appliance lock-in.

Microsoft Azure Stack HCI

Licence: Microsoft subscription + per-core fee. Architecture: Hyper-V + Storage Spaces Direct + Azure Arc. Best for: Microsoft-aligned shops with Azure relationship. Strengths: Strong Microsoft ecosystem integration. Limits: Locks into Microsoft licensing economics.

Oracle Cloud Native Environment / Oracle Linux Virtualization Manager

Licence: Subscription / commercial. Best for: Oracle-aligned organizations.

Sovereign hyperscaler regions

AWS Sovereign Cloud (EU / US Gov)

Dedicated regions with sovereignty controls. Some satisfy member-state mandates; others don’t, depending on jurisdiction.

Azure Sovereign Cloud (Azure Government, Azure Germany historically)

Similar pattern.

GCP Sovereign Cloud

GCP’s sovereign offerings, Workspace partnerships in some EU markets.

Trade-off: these provide cloud-managed convenience but leave the service plane under hyperscaler control. For substantive sovereignty (encryption keys customer-controlled, supplier transparency, exit-readiness), customer-owned infrastructure typically wins.

Regional cloud providers (private cloud as-a-service)

A growing market in 2026:

• Hetzner (Germany) — bare metal + cloud, popular in DACH
• OVHcloud (France) — strong EU sovereign positioning
• Regional providers running Ænix Public Cloud Platform — hosting providers that sell a sovereign cloud product built on Cozystack
• Various regional providers per jurisdiction

These offer private-cloud-style isolation without you operating the platform. Trade-off: provider relationship vs. direct hardware control.

How to choose

Decision tree:

1. Need multi-tenant + open-source + Kubernetes-native + sovereignty? → Cozystack.
2. Existing VMware estate, financial-services renewal pressure? → Plan VMware exit. Destination: typically Cozystack or OpenShift.
3. OpenStack expertise + large telco / government scale? → OpenStack remains valid.
4. Existing Red Hat / OpenShift commitments? → OpenShift Virtualization.
5. SMB / single-tenant? → Proxmox VE.
6. Don’t want to operate the platform yourself? → Regional sovereign cloud provider (Hetzner, OVHcloud, or a regional provider running Ænix Public Cloud Platform).
7. AI/GPU at scale, sustained utilization? → Cozystack or OpenShift on dedicated GPU infrastructure.
8. Sovereignty + EU + low operational footprint? → Cozystack with Ænix support, or OVHcloud.

Migration paths

Most modern private-cloud deployments are migrations from existing infrastructure:

• VMware → Cozystack/OpenStack/OpenShift — KubeVirt-based migration, image conversion, multi-tenancy redesign
• Public cloud → private cloud (repatriation) — workload classification, cost honesty, destination architecture; see Cloud repatriation
• OpenStack → Cozystack — for teams seeking Kubernetes-native foundation; image migration is straightforward
• Hyperscaler region → sovereign region — for sovereignty-driven migrations within hyperscaler model

Want to dig deeper?

• Private cloud platform — Cozystack
• Private cloud consulting — engineering services
• VMware alternative — VMware exit
• Cloud repatriation — public cloud exit
• cozystack.io — open-source project

Private cloud has moved from “yesterday’s architecture” to “tomorrow’s default for regulated and cost-sensitive workloads” within ~3 years. The Broadcom Private Cloud Outlook 2025 found 53% of organizations now prioritize private cloud for new workloads. The LSEG Global Cloud Survey reports 84% of financial-services firms have adjusted cloud strategy due to regulatory pressure. The shift is real.

But the architecture decisions in 2026 are different from 2018-era OpenStack-centric private clouds. The default stack has moved to Kubernetes-native virtualization (KubeVirt) plus open-source storage and networking. The operational model has matured. The trade-offs are clearer.

What “private cloud” means in 2026

A private cloud is dedicated cloud infrastructure run for a single organization or tenant, with self-service consumption patterns matching public cloud (provision-on-demand, multi-tenancy, observability, automation) — but on infrastructure the organization controls.

In 2026, private cloud is one of:

• Customer-owned, customer-operated — organization owns hardware, runs the platform.
• Customer-owned, vendor-operated — organization owns hardware, vendor runs the platform under contract.
• Vendor-owned dedicated — vendor provides dedicated infrastructure (single-tenant), organization consumes.
• Sovereign hyperscaler region — hyperscaler-operated infrastructure with sovereignty controls (some count, some don’t).

This article focuses on customer-owned options 1 and 2 — the architecture is similar; the operations differ.

Architectural layers

A modern private cloud has six functional layers:

Layer 1: hardware

• Compute — x86 or ARM servers; modern generations support all relevant workloads. AI/GPU adds a separate hardware tier (NVIDIA H100/H200/L40S/Blackwell, AMD MI-series).
• Storage — replicated block storage (LINSTOR, Ceph, or vendor SAN). Object storage for backup and applications.
• Network — datacenter fabric (BGP-routed leaf-spine increasingly default), 25-100 Gbps Ethernet sufficient for most non-HPC workloads.
• Datacenter — own facility, colocation, or ROBO/edge. Power, cooling, physical security.

Layer 2: OS and platform foundation

• OS — Linux, increasingly minimal (Talos, Bottlerocket, Flatcar) for Kubernetes hosts. RHEL / Ubuntu LTS for VM hypervisors when not in Kubernetes.
• Kubernetes — vanilla, OpenShift, Cozystack, vendor-led. Distribution choice is structural.
• Virtualization — KubeVirt for VM workloads inside Kubernetes (most modern); KVM/libvirt directly (OpenStack); VMware (legacy).

Layer 3: storage and networking

• Block storage — LINSTOR (DRBD-based, default in Cozystack), Ceph (Rook-managed), vendor SAN.
• Object storage — Ceph RGW, MinIO, SeaweedFS.
• Network virtualization — Cilium (eBPF, default in 2026), Calico, OVN.
• Load balancing — MetalLB for layer 2/3, Cilium L7, ingress controllers (NGINX / Traefik / Contour).

Layer 4: control plane

• Multi-tenancy — Tenant CRD (Cozystack), namespace-based (vanilla Kubernetes), vCloud-Director-style (legacy).
• Identity — Keycloak, Okta integration, AD integration. SPIFFE/SPIRE for service identity.
• GitOps — Argo CD or Flux. Cozystack uses Flux.
• Observability — VictoriaMetrics + VictoriaLogs (Cozystack default), Prometheus + Loki, vendor SaaS.
• Backup/DR — Velero + S3, per-app patterns (PostgreSQL PITR).

Layer 5: application and platform services

• Managed databases — PostgreSQL (CloudNativePG), MariaDB, MongoDB, Redis, Valkey, Kafka, ClickHouse, RabbitMQ, NATS, OpenSearch.
• Object storage as a service — S3-compatible.
• AI/ML platform — KubeVirt for VM-based GPU, Kubernetes-native for container-based GPU, vLLM/Triton for inference.
• Self-service portal — Backstage, Cozystack Dashboard, custom.

Layer 6: operations

• Runbooks — documented operational procedures.
• On-call rotation — incident response model.
• Capacity planning — quarterly hardware refresh, growth forecasting.
• Compliance posture — audit logging, certifications (where applicable), regulator dialog.

Three architectural patterns that work

Pattern 1: Cozystack-based Kubernetes-native cloud

What: Single Kubernetes cluster (or fleet) with Cozystack as the platform layer. KubeVirt for VMs, Kubernetes for containers, LINSTOR for storage, Cilium for networking, Tenant CRD for multi-tenancy, Cozystack Dashboard for self-service.

Best for: Service providers, sovereign-cloud builders, regulated multi-tenant. Greenfield deployments.

Pros: Open-source, single-stack, multi-tenancy native, virtualization + container in one platform.

Cons: Newer than OpenStack; smaller community than Kubernetes-only deployments.

Pattern 2: OpenStack-based traditional private cloud

What: OpenStack as compute orchestrator, Ceph for storage, OVN for networking, Heat/Terraform for provisioning. Optional Kubernetes-on-OpenStack for container workloads.

Best for: Organizations with deep OpenStack experience; large telco/sovereign deployments where OpenStack is the procurement standard.

Pros: Mature, broad community, many vendor options.

Cons: Operationally complex; harder to find OpenStack engineers in 2026; less Kubernetes-native.

Pattern 3: VMware Cloud Foundation (VCF) — legacy

What: vSphere + vSAN + NSX + vCD + Aria (formerly vRealize). Closed source, subscription-licensed.

Best for: Existing VMware estates that haven’t yet been triggered out by Broadcom economics.

Pros: Mature, well-known, extensive integration.

Cons: Subscription-only post-Broadcom, 2-5× price increases observed, lock-in to single vendor’s roadmap.

(For VMware exit guidance see VMware alternative.)

Architectural decisions that matter most

Five decisions with the highest long-term impact:

Decision 1: virtualization layer

KubeVirt or traditional hypervisor? KubeVirt is the 2026 default for greenfield; traditional hypervisor (KVM/libvirt directly) is appropriate for very-large-scale OpenStack deployments where KubeVirt’s overhead matters.

Decision 2: storage architecture

LINSTOR/DRBD for replicated block? LINSTOR is operationally simpler and Cozystack-default. Ceph is more flexible but operationally heavier. Vendor SAN is an option but contracts limit your scaling pattern.

Decision 3: networking

Cilium has become the default CNI for new deployments. NSX-equivalent (Cilium L7, service mesh) replaces VMware NSX functionality. The decision is more about your team’s eBPF familiarity than about technical fit.

Decision 4: multi-tenancy model

For service-provider model: Tenant CRD (Cozystack) is the default. For internal multi-BU: namespace-based + RBAC is sufficient. For absolute isolation: cluster-per-tenant (operationally expensive).

Decision 5: operational model

Customer-operated (you run it), vendor-operated (Ænix or similar runs it for you), or hybrid (you operate; vendor provides 2nd-line). Decision driven by internal team capacity and risk appetite.

Capacity sizing

A practical sizing rubric for a 100-VM-equivalent workload:

• Compute: 6-10 servers, dual-socket, 256-512 GB RAM each. Reserve 30% headroom for failures and growth.
• Storage: ~100 TB replicated (3-replica) for steady-state plus snapshot/backup capacity. ~300 TB raw for that target.
• Network: 25 Gbps NIC per server, leaf-spine fabric, 100 Gbps backbone.
• Datacenter: ~6-10 kW per rack at modern density.
• DR site: comparable secondary footprint.

Scaling beyond 1000 VMs: hardware grows roughly linearly; platform team grows sublinearly with good automation.

Operational practices that matter

• Quarterly capacity reviews — hardware ahead of growth.
• Twice-yearly Kubernetes upgrades — stay current on CVEs and feature parity.
• Documented incident response — incident commander, scribe, blameless post-mortems.
• Compliance posture — audit logs, certifications, regulator dialog where applicable.
• Platform-team headcount sized realistically — 1-3 engineers for a single-cluster small private cloud; 5-15 for a multi-cluster service-provider.

Common architectural mistakes

Mistake 1: copying public-cloud architecture

Designing the private cloud to look like AWS internally. Different scale economics; public-cloud architecture patterns (massive distributed eventual-consistency systems) are over-engineering for most private clouds.

Mistake 2: vendor-led private cloud

Buying “private cloud appliance” rebuilds lock-in with new vendor. The vendor’s roadmap becomes yours.

Mistake 3: under-engineering operations

Compute and storage solved; observability/identity/backup-DR/runbook layer underinvested. Operational debt builds.

Mistake 4: skipping multi-tenancy at design

Single-tenant cluster scaled to multi-tenant later. Bolted-on multi-tenancy fails at scale or under regulator audit.

Mistake 5: hardware refresh skipped in budget

Year 4 hardware refresh ignored in initial economics. The refresh cliff arrives unexpectedly.

Want to dig deeper?

• Private cloud consulting services — engagement details
• Data sovereignty — sovereignty trigger
• Cloud repatriation — when coming from public cloud
• VMware alternative — when coming from VMware
• Cozystack — open-source platform foundation

The terms platform engineering, DevOps, and SRE have been used interchangeably, in opposition, and as overlapping practices for nearly a decade. By 2026 the industry has roughly converged — but only roughly. Different companies still use the same words for different jobs, and the resulting org-design conversations stall because nobody quite agrees on what they’re discussing.

This is the working version we use at Ænix when we engage with a customer’s engineering organization.

The three terms in one paragraph each

DevOps is a cultural and operational practice within product teams. Its premise: the same team that builds the software is responsible for operating it in production. Tooling supports this — CI/CD pipelines, IaC, observability — but tooling alone is not DevOps. The function lives inside product teams, not as a separate department.

SRE (Site Reliability Engineering) is a discipline that applies software engineering to operations. The original Google formulation: SREs are software engineers tasked with keeping production reliable, with explicit error budgets, SLOs, and a rule that operational toil cannot exceed 50% of their time. SRE can sit inside product teams (embedded SRE) or as a separate function (centralized SRE).

Platform engineering is the practice of building and operating internal platforms that product teams consume. Its premise: don’t expect every product team to figure out infrastructure, observability, security, identity, and release engineering on their own. Build a self-service platform that handles those concerns, and let product teams ship. Platform engineering lives as a separate function — a team whose customers are other engineering teams.

Where they overlap

All three care about:

• Reliability — SLOs, error budgets, incident response.
• Tooling — CI/CD, IaC, observability, secret management, identity.
• Velocity — making it faster for product teams to ship.
• Operational excellence — automation, runbooks, post-mortems.

Where they diverge: who owns what, and whose problem is it when the abstraction breaks.

Where they don’t overlap

The cleanest way to remember: DevOps is a practice, SRE is a discipline, platform engineering is a function. They can coexist; they answer different questions; they’re not in opposition.

When each is appropriate

DevOps without separate SRE or platform engineering

Small to mid-size organizations with 1-3 product teams. Each team operates its own services, with shared tooling but no dedicated reliability or platform function. The model works because each team can fully understand its own operational surface.

DevOps + SRE without platform engineering

Mid-size organizations with reliability concerns that justify a dedicated function. Common in mid-stage startups where production reliability is becoming critical but the engineering organization is still small enough that infrastructure decisions don’t fragment.

DevOps + platform engineering without separate SRE

Mid-size to large organizations where the leverage is in making infrastructure self-service rather than in keeping individual services reliable. SRE-style practices (SLOs, error budgets) get baked into the platform; reliability concerns are addressed at platform level rather than per-service.

All three

Large organizations with both reliability requirements and self-service platform needs. SREs work on reliability of critical services; platform engineering provides the substrate; product teams practice DevOps within the constraints both establish.

The mistake to avoid: hiring a “DevOps engineer” who actually does platform engineering work, or hiring an “SRE” who actually does general infrastructure. The titles increasingly mean specific things.

What platform engineering actually builds

For organizations that need a platform engineering function, the question becomes: what does the platform team actually produce? Concretely:

1. Internal developer platform (IDP)

A self-service surface that product teams use without filing tickets. Common scope:

• Environment provisioning (dev / staging / prod)
• Application deployment (Helm / Argo CD / Flux)
• Observability onboarding (metrics / logs / traces)
• Secrets management
• Identity (workforce identity → service identity)
• Network connectivity (mesh / ingress / load balancing)

The IDP is consumed via a UI (often Backstage or similar), via APIs, or via IaC. The choice depends on the team’s preferences — but the underlying capabilities must be self-service regardless of UI.

2. Golden paths

Opinionated, well-supported paths for common product-team needs. “Golden” because they’re the recommended way; teams can deviate but pay for it in support effort. Examples:

• Standard service template (HTTP API, batch job, scheduled job)
• Standard deployment pattern (canary, blue-green)
• Standard observability stack (auto-instrumented)
• Standard data-access patterns (managed databases via the platform)

3. Operational model

The platform itself is operated by the platform team. SLOs, on-call, runbooks, incident response — all platform-team responsibilities. Product teams should be able to assume the platform works, not contribute to fixing it.

4. Internal product management

Platform engineering as a function has product-team customers. That requires product-management practices: roadmap, prioritization, customer interviews (your engineering teams), feedback loops, deprecation policies.

This is where many platform teams underperform. Engineering excellence without product orientation produces a platform nobody adopts.

Tools that matter

Platform engineering tools fall in five buckets:

Compute and orchestration

Kubernetes is the de facto orchestration layer for most modern platforms. Distributions: vanilla, OpenShift, Rancher, Cozystack (open-source Kubernetes-native virtualization), vendor-led variants. The choice is largely determined by the operational model.

Infrastructure-as-Code

Terraform, OpenTofu, Pulumi — for cloud and infrastructure provisioning. Crossplane — for Kubernetes-native infrastructure abstraction. Choice depends on team familiarity and target environments.

GitOps

Argo CD and Flux are the two production-grade GitOps engines. Both work; Flux is closer to the upstream Kubernetes way; Argo CD has stronger UI ergonomics. Cozystack uses Flux as the default.

Internal developer portal

Backstage (CNCF Incubating) is the dominant choice. Alternatives: Port, Cortex, Compass, custom. Important note: Backstage is a portal (catalog + UI), not a platform. The platform sits underneath; the portal exposes it.

Observability

VictoriaMetrics + VictoriaLogs (open-source, low-overhead — Cozystack default), Prometheus + Loki, or commercial (Datadog, New Relic). Self-hosted matters for sovereignty-sensitive workloads.

Secrets and identity

External Secrets Operator, HashiCorp Vault, customer-controlled HSMs. Workload identity via SPIFFE/SPIRE for service-to-service auth.

The toolset is converging. The differentiation is in how these tools are wired together as a coherent platform — that’s where platform engineering earns its keep.

Metrics that matter

For evaluating a platform engineering function:

• Time-to-environment — from product-team request to a working environment. Target: hours, not weeks.
• Golden-path adoption — fraction of new services using the recommended template / deployment / observability pattern.
• Platform-team-to-product-engineer ratio — typically 1:10 to 1:20 in mature organizations.
• Internal NPS / customer satisfaction — from product teams about the platform.
• Toil ratio — fraction of platform-team time spent on tickets vs. golden-path work. Target: keep below 50%.
• Platform availability SLO — yes, the platform itself has an SLO. Production-grade is 99.9%+ for IDP services.

DORA metrics (deployment frequency, lead time, change-failure rate, time-to-restore) are product-team metrics — but the platform’s job is to make those metrics achievable.

Common pitfalls (for the platform team)

Pitfall 1: building for engineers, not for product teams

The platform team’s customers are product teams, not other platform engineers. The architecture should optimize for the product team’s experience, not for engineering elegance.

Pitfall 2: Backstage as the platform

Backstage is an excellent IDP frontend. It’s not a platform. Buying Backstage without an underlying opinionated platform produces a catalog over the same operational mess.

Pitfall 3: too many opinions, too rigid

Golden paths should be golden, not gold-plated. Product teams need to be able to deviate when their case is special. The platform’s authority comes from being genuinely useful, not from policy enforcement.

Pitfall 4: under-staffed

Platform teams that absorb both platform-build and on-call duties for shared services tend to spend their time on tickets. Capacity for golden-path work disappears. The function stalls.

Pitfall 5: vendor-led platform-in-a-box

Several vendors sell “complete platform engineering solutions.” These work for narrow customer profiles but rebuild the lock-in problem. The vendor’s roadmap becomes your roadmap.

A maturity progression

A typical organization moves through these stages:

1. Pre-platform — each team owns its own infrastructure. Tooling is fragmented. Shared services emerge ad-hoc.
2. Shared infrastructure — central team owns shared infrastructure (Kubernetes, CI/CD, observability). No self-service yet; teams file tickets.
3. Self-service primitives — central team exposes APIs / IaC modules. Teams self-serve common operations. Limited golden paths.
4. Internal developer platform — opinionated platform with golden paths. Most operations are self-service. Platform team has product-team customers.
5. Mature platform engineering — platform is a coherent product. SLOs, internal product management, customer-driven roadmap, controlled deprecation.

Most organizations sit between stages 2 and 3. The leverage of moving to stage 4 is large but requires intentional investment.

When to engage external help

External platform engineering help is the right call when:

• A specific deadline (regulator, board mandate, scaling moment) makes internal-only build too slow.
• Your team has the long-term capacity but lacks specific senior platform-engineering experience to start cleanly.
• You want a structured external assessment to validate or challenge internal direction.
• A repatriation or migration program needs platform engineering velocity that exceeds internal hiring rate.

It’s the wrong call when:

• The engineering organization isn’t ready (no platform-team headcount, no internal customer, no problem clearly named).
• The decision is already made and the engagement is meant to validate it.

Want to dig deeper?

• Platform engineering services page — engagement details
• Internal developer platform — IDP-specific engagement
• Platform Readiness Assessment — assessment methodology
• Cozystack — the platform we typically build on

OpenStack remains widely deployed in telecom and government infrastructure. It also faces structural pressure: shrinking pool of OpenStack engineers, operational complexity that grows with deployment age, and competition from Kubernetes-native alternatives that didn’t exist when OpenStack was designed.

Where OpenStack still wins

• Telecom-scale deployments — running thousands of nodes with telecom-specific features (NFV, DPDK integration, SR-IOV, high-throughput networking). Established expertise; vendor distros mature.
• Government / sovereign clouds — large public-sector OpenStack deployments where OpenStack is the procurement-mandated platform.
• Existing investments — organizations 5+ years into OpenStack with deep expertise. Migration cost may exceed continuing operations cost.
• Specific features — some OpenStack capabilities (deep network programmability, specific telco features) don’t have direct Kubernetes equivalents yet.

Where the pressure is

• Engineer scarcity — OpenStack expertise pool is shrinking. New engineers are trained on Kubernetes, not OpenStack.
• Component sprawl — 30+ services in a typical OpenStack deployment, each with its own lifecycle, upgrade cadence, integration tests.
• Upgrade pain — major-version OpenStack upgrades remain operationally heavy.
• Vendor distro fragmentation — Red Hat OSP, Mirantis, Canonical, others — each with different opinions and support models.

Modernization paths

For OpenStack operators considering modernization:

Path 1: stay and tune

Continue OpenStack; invest in operational practices (Helm-based deployments, GitOps, automation). Right when OpenStack expertise is deep and migration cost exceeds value.

Path 2: Kubernetes-on-OpenStack

Run Kubernetes platforms (Cozystack or other) on top of OpenStack as a tenant. Adds another platform layer; some teams find this practical for gradual transition.

Path 3: parallel deployment

Build Cozystack alongside OpenStack on new hardware. Migrate workloads cohort by cohort. Decommission OpenStack as cohorts complete. Most common path for full modernization.

Path 4: full lift-and-shift to Kubernetes

Aggressive — replace OpenStack control plane with Kubernetes-based equivalent (Cozystack). Higher risk; faster outcome.

Cozystack architecture for OpenStack-trained teams

Some helpful translations:

Most OpenStack engineers find Cozystack’s operational model simpler — fewer moving parts, more declarative, integrated observability.

Practical migration approach

For mid-size (50-500 hosts) OpenStack to Cozystack:

1. Assessment (14 or 28 days) — current OpenStack deployment, workload classification, target Cozystack architecture.
2. Cozystack foundation — parallel deployment on new or repurposed hardware, overlapping the first cohorts.
3. Migration cohorts — workloads move cohort by cohort. Image migration via KVM→KubeVirt.
4. OpenStack decommission — staged as cohorts complete.

Total elapsed: 4-12 months for a mid-size deployment; 12-18 months with complex provider networks or tenant-facing OpenStack APIs.

OpenStack remains widely deployed in telecom and large-enterprise infrastructure. Modernization is not a one-size-fits-all conversation: mature telco-scale OpenStack with deep vendor distro support is a different migration than a mid-size enterprise running upstream OpenStack with a small ops team. Both can move to Cozystack; the phasing and risk profile differ substantially.

Where OpenStack still works (and we’ll say so)

Before discussing migration, the honest reverse case. OpenStack remains the right answer for:

• Tier-1 telco NFV environments where vendor distro (Red Hat OSP, Mirantis, Canonical, Wind River) still has support runway and VNF certification is OpenStack-specific
• Very large estates (>1,000 nodes) with deep OpenStack expertise where operational complexity is already amortised
• Government clouds where OpenStack is the procurement-mandated platform (some EU member-state and APAC public-sector tenders)
• Existing investments at year 2-3 of a 5-year deployment where migration cost would exceed continuing operations cost

For everyone else, modernization typically warrants serious evaluation.

What pushes OpenStack-running organisations to migrate

Three pressures dominate the 2026 conversation:

1. Vendor distro lifecycle

Red Hat is steering OSP (OpenStack Platform) customers toward OpenShift-based offerings. Other distributions (Mirantis, Canonical) remain on the market, but the vendor landscape for OpenStack distributions is consolidating. Check the support dates in your own vendor contract: for many mid-2020s deployments they fall within the next few years.

2. Engineer scarcity

OpenStack expertise is a shrinking pool. New engineers are trained on Kubernetes, not on OpenStack’s Nova/Neutron/Cinder/Keystone component model. Operators with deep OpenStack expertise are aging out or moving to other roles. Hiring is harder; retention is harder.

3. Service-catalog ceiling

OpenStack’s primary scope is IaaS (compute, network, storage). Managed databases (Trove), container orchestration (Magnum), and modern service families (managed Kafka, S3-equivalent at scale, GPU-as-a- service, AI inference) either bolt on awkwardly or live outside the platform entirely. For operators whose customers or internal teams increasingly expect platform-grade services, OpenStack falls behind without significant additional engineering.

Component mapping

For OpenStack operators evaluating Cozystack, the canonical translation:

Most OpenStack engineers find Cozystack’s operational model simpler once they cross the Kubernetes learning curve — fewer moving parts, more declarative, integrated observability. The learning curve itself is real: 4-8 weeks per engineer for focused training, longer for domain depth.

Cohort-based migration phases

Phase 0 — Assessment (14 or 28 days)

Inventory the OpenStack deployment:

• Service-by-service usage — which OpenStack services do you actually use (Nova / Neutron / Cinder almost always; everything else is variable)
• Workload inventory — instance count, OS mix, vCPU/RAM/disk profiles, criticality tier
• Network inventory — subnets, security groups, floating IPs, external network configurations, Octavia load balancers
• Storage inventory — Cinder volume types, snapshot retention, Swift bucket usage
• Multi-tenancy — project / domain hierarchy, custom roles, RBAC policies
• Integrations — vendor VNF certifications, observability hooks, CI/CD pipelines, ITSM tooling

Output: migration plan with workload buckets (migrate-now / migrate-later / stay / re-architect), risk flags, phasing options.

Phase 1 — Cozystack foundation

Hardware procurement (or repurpose of OpenStack-freed capacity in later phases). Cozystack platform deployed on new hardware in parallel to existing OpenStack. Cilium networking validated against OpenStack network configurations expected to translate. LINSTOR storage operationalised at scale. Federated identity (Keycloak + customer IdP).

Tenant CRD model designed to map cleanly from OpenStack project / domain hierarchy. Each top-level OpenStack project typically becomes a Cozystack Tenant; nested projects become nested Tenants.

End state: Cozystack platform up, internally validated, ready for workload onboarding.

Phase 2 — Operational tooling

Observability stack (VictoriaMetrics + VictoriaLogs) integrated with customer SIEM. Backup/DR with Velero + per-app patterns. Runbook library. On-call rotation. Incident response process. GitOps deployment workflow (Flux default; Argo CD if customer prefers).

End state: production-operations-grade tooling in place; team training in progress.

Phase 3 — Workload migration cohorts

Cohorts of 50-200 instances migrating at a time. Per cohort:

1. Image conversion — OpenStack Glance images converted to KubeVirt-compatible format. Most KVM-based OpenStack images convert with qemu-img convert + minor metadata adjustment. Windows instances get virtio driver injection.
2. Network mapping — OpenStack subnets to Cilium ClusterPool + NetworkPolicies, security groups to NetworkPolicies, Octavia load balancers to MetalLB + ingress controllers.
3. Storage migration — Cinder volume data migrated to LINSTOR. Snapshot history preserved or pruned per retention policy. For Swift-stored data, S3 API compatibility allows direct migration to SeaweedFS.
4. Validation window — workload runs in parallel on Cozystack for typically 7-14 days. Application owner sign-off required before final cutover.
5. Final cutover — DNS / load balancer flip to Cozystack endpoint. OpenStack instance kept available for 7-30 days rollback window.

Phase 4 — Operational handover (parallel to Phase 3)

Ænix engineers reduce direct involvement. Customer operations team absorbs first- and second-line incidents. Ænix support (Plus or Enterprise tier for 24×7) continues for escalation. Documentation handover. Knowledge transfer sessions.

Phase 5 — OpenStack decommission

As migration cohorts complete, OpenStack capacity is repurposed into the Cozystack cluster. Hardware is the same commodity x86; OpenStack software stack is retired. Vendor distro contracts wound down per their lifecycle.

Where OpenStack-to-Cozystack migrations stumble

1. Network model redesign

OpenStack Neutron’s network model has historically been highly configurable — provider networks, tenant networks, security groups, floating IPs, FWaaS, VPNaaS, complex routing topologies. Cilium’s eBPF model is fundamentally different: L4/L7 NetworkPolicies, eBPF- based routing, Hubble for observability.

Translating elaborate Neutron configurations to Cilium requires careful architecture work in Phase 0-1. Skipping this produces production incidents in Phase 3 where customer workloads expected specific network behaviour that doesn’t translate directly.

2. Multi-tenant policy translation

OpenStack’s role-based access (Keystone roles, project policies) doesn’t map 1:1 to Kubernetes RBAC + Tenant CRD. Custom roles defined for specific OpenStack APIs may not have direct Kubernetes equivalents. Plan time for policy redesign rather than mechanical translation.

3. VNF certification

Tier-1 telco deployments running certified VNFs face a different challenge: the VNF vendor certifies on specific OpenStack distros, not on Cozystack. Three approaches:

• Run VNFs on KubeVirt under Cozystack — the VNF runs as a VM; vendor certification may or may not extend to this configuration. Vendor dialogue required.
• Keep certified VNFs on OpenStack — parallel platforms for the lifecycle of the certified VNF; modernization track for new VNFs on Cozystack.
• Negotiate cloud-native equivalent — many VNF vendors are migrating toward Cloud-Native Network Functions (CNFs) on Kubernetes; the migration may align with vendor’s own CNF modernization.

In practice all three patterns appear, often in the same operator, depending on which VNF vendor and which generation.

4. Operational culture shift

OpenStack operators are used to imperative APIs (CLI commands, REST calls, console actions). Cozystack expects GitOps for production changes. This is a culture shift, not just a tool shift. Engineers who built OpenStack expertise around imperative workflows need 4-8 weeks of focused training plus 3-6 months of practice to internalise the GitOps discipline.

Ænix engagement includes training as a workstream; customer investment in the cultural transition is also required.

Timeline realities

Mid-size enterprise (200-500 nodes, simple multi-tenancy, mostly default networking):

• Phase 0: 14 or 28 days
• Phases 1-2: foundation and tooling, overlapping the first cohorts
• Phase 3: the bulk of the elapsed time
• Phases 4-5: alongside the last cohorts

Total: 4-12 months for a mid-size deployment; 12-18 months with complex provider networks or tenant-facing OpenStack APIs

Tier-1 telco (1,000-5,000 nodes, complex multi-tenancy, certified VNF environments, NFV-specific networking):

• Phase 0: 2-3 months
• Phase 1: 4-6 months
• Phase 2: 2-3 months
• Phase 3: 12-24 months (multiple parallel cohorts)
• Phase 4-5: 6-12 months
• VNF modernization track: 18-36 months in parallel

Total: 24-48 months for full modernization; first production workloads on Cozystack within 12-18 months

When OpenStack-to-Cozystack migration is the right answer

Strong fit:

• Vendor distro lifecycle is forcing a decision within 24 months
• Engineer scarcity is starting to hurt operational quality
• Customer demand for managed databases / S3 / containers / GPU services exceeds what OpenStack natively handles
• Modernisation budget is available across 2-4 years

Marginal fit:

• Stable, mature OpenStack deployment with deep team expertise and no service-catalog pressure — modernisation can wait for vendor lifecycle
• Very large estates (>5,000 nodes) where modernisation cost is in the multi-tens-of-millions range — phased multi-year programme required

Poor fit:

• Recently-deployed OpenStack at year 1 of a 5-year programme — finish what you started; modernise at lifecycle
• OpenStack-on-government-procurement-mandate without flexibility to change platforms

Engagement structure

• Discovery call (30 min, free)
• Platform Readiness Assessment (fixed price, 14 days focused or 28 days full) — workload buckets, phasing options, risk flags
• Pilot deployment (2-3 months) — Cozystack stood up, 50-100 workloads migrated, billing / operational workflows validated
• Cohort migration — workload migration in cohorts; 4-12 months in total for a mid-size deployment, 12-18 months with complex provider networks or tenant-facing OpenStack APIs, longer at telco scale
• OpenStack decommission (parallel to cohort migration) — staged as cohorts complete
• Support subscription (ongoing) — Plus or Enterprise support tier for 24×7 escalation (see /pricing/)

Where to dig deeper

• OpenStack migration hub — commercial landing
• OpenStack vs Cozystack modernization — the modernization path analysis
• OpenStack alternative — alternative-focused commercial landing
• Public Cloud Platform product page — common target product for hosting-provider OpenStack migrations

OpenShift Virtualization (Red Hat) and Cozystack (Ænix / CNCF Project) are the two most-mature KubeVirt-based platforms in 2026. They share architectural foundations but differ in commercial model, operational footprint, and vendor relationship.

Shared foundations

Both platforms run KubeVirt for VM workloads alongside Kubernetes containers. Both inherit Kubernetes operational patterns (declarative config, GitOps, RBAC, observability). Both support production VM workloads with live migration, snapshots, and multi-tenant isolation.

Where they differ

Commercial model

OpenShift Virtualization: Red Hat commercial subscription. Per-core or per-socket pricing. Includes Red Hat support, certification, and ecosystem access.

Cozystack: Apache 2.0 open source. Ænix offers commercial support tiers; you can self-deploy with no commercial relationship.

For organizations standardized on Red Hat procurement, OpenShift is administratively simpler. For organizations seeking open-source-first or service-provider economics, Cozystack fits better.

Operational footprint

OpenShift: broad surface area — OpenShift Container Platform plus Virtualization plus Service Mesh plus Pipelines plus other addons. Operationally rich; team needs OpenShift-specific expertise.

Cozystack: focused stack — KubeVirt + Cilium + Kube-OVN + LINSTOR + Cozystack Dashboard + observability. Lighter operational footprint; standard Kubernetes operational expertise transfers directly.

Multi-tenancy

OpenShift: namespace-based with Project CRD; RBAC and quotas at namespace level. Works for enterprise multi-BU.

Cozystack: Tenant CRD with nested tenants, scoped audit, billing-friendly model. Works for service-provider multi-customer plus enterprise multi-BU.

Vendor relationship

OpenShift: Red Hat / IBM relationship. Roadmap shaped by Red Hat’s commercial decisions.

Cozystack: open-source community-governed (CNCF Project). Ænix is the largest contributor but not the owner. Roadmap shaped by community + commercial users.

Ecosystem integration

OpenShift: integrates with broader Red Hat ecosystem (Ansible, Satellite, Identity Management, etc.).

Cozystack: integrates with broader CNCF ecosystem (Prometheus stack alternatives, Argo, Crossplane, etc.).

When OpenShift wins

• Existing Red Hat / OpenShift commitments
• Enterprise procurement standardized on Red Hat
• Need integrated Red Hat ecosystem (Ansible Automation Platform, etc.)
• Procurement that requires a Red Hat contract specifically

When Cozystack wins

• SLA-backed support from the maintainers — published response times, 24×7 on the Plus and Enterprise tiers (pricing); AENIX s.r.o. holds ISO/IEC 27001
• Open-source-first procurement
• Service-provider model (multi-customer cloud)
• Sovereignty / regulator requirements where open-source matters
• Cost-sensitive at scale (no per-core subscription)
• Greenfield with no existing Red Hat relationship
• Need lighter operational footprint than full OpenShift

Migration between them

Both KubeVirt-based, so VM-level migration is straightforward (image-level compatibility). The architectural delta is in:

• Multi-tenancy model (Project CRD vs Tenant CRD)
• Networking (OVN-Kubernetes, which replaced the deprecated OpenShift SDN, vs Cilium)
• Storage (OpenShift Data Foundation / Ceph vs LINSTOR / DRBD)
• Operational tooling (OpenShift CLI/Console vs Cozystack Dashboard/standard kubectl)

Realistic migration timeline: 3-9 months for mid-size deployment.

How to choose

If your situation matches OpenShift’s strengths (Red Hat ecosystem, enterprise procurement, broad capability surface) — OpenShift Virtualization. If your situation matches Cozystack’s strengths (open-source, service-provider, lighter footprint) — Cozystack.

For a specific evaluation, the assessment phase of either engagement helps clarify. See Platform Readiness Assessment.

In 2026 the realistic shortlist for production virtualization platforms includes (among others) Nutanix AHV, VMware Cloud Foundation, and Cozystack. Each represents a different architectural philosophy.

Architectural philosophies

Nutanix: vendor-led integrated HCI appliance. Operational simplicity and integrated support are the core value proposition.

VMware: mature legacy stack with deep ecosystem integration. Vendor-managed roadmap; subscription-led economics.

Cozystack: open-source Kubernetes-native platform. Customer-controlled architecture; community-governed roadmap.

Detailed comparison

When each wins

Nutanix wins

• Existing Nutanix HCI investment with operational expertise
• Strong preference for integrated appliance + commercial support
• VM-only workload portfolio
• Mid-size enterprise with integrated procurement

VMware wins

• Existing VMware estate where renewal economics are still tolerable
• Deep vSphere expertise that’s hard to migrate
• Specific VMware-only features (some niche advanced networking, storage)
• (Increasingly rare in 2026 due to Broadcom pricing)

Cozystack wins

• Service-provider or multi-tenant cloud-builder model
• Sovereignty / regulator requirements
• Open-source procurement preference
• Mixed VM + container workloads on one platform
• AI/GPU at scale with Kubernetes-native tooling

Migration economics

Migrating between these platforms is not free. Realistic cost estimates:

• VMware → Cozystack: about 8-12 months for ~100 VMs and 18-24 months for ~1,000 VMs, including planning and migration waves; assessment + destination build + cohort migration. Net positive after Year 2 typically.
• VMware → Nutanix: Similar timeline; uses Nutanix Move tooling.
• Nutanix → Cozystack: full-estate migration typically 9-18 months, by scope; KVM image compatibility helps.
• Cozystack → VMware/Nutanix: Rare in 2026 (reverse migration).

How to decide

The decision tree:

1. Existing platform with deep expertise + economics still work? → Stay.
2. Multi-tenant or service-provider model? → Cozystack.
3. Sovereignty / open-source-first procurement? → Cozystack.
4. HCI appliance preference + Nutanix relationship? → Nutanix.
5. VMware estate, no triggers to leave? → VMware (with eye on next renewal).
6. Greenfield + Kubernetes-fluent team? → Cozystack.

The Network and Information Security Directive 2 (Directive (EU) 2022/2555 — NIS2) replaced the original NIS Directive in 2023. Transposition into national law was due by 17 October 2024. Some EU member states completed transposition on time; others ran late. Either way, by mid-2025 NIS2 is operational across the EU, with competent authorities in each member state and the European Cybersecurity Agency (ENISA) playing a coordination role.

For in-scope entities — and the ICT third parties serving them — the architectural work is concrete, even if the regulatory text is broad.

Who is in scope

NIS2 applies to two categories of entities, with sectoral scoping:

Annex I — sectors of high criticality

• Energy (electricity, gas, oil, district heating/cooling, hydrogen)
• Transport (air, rail, water, road)
• Banking
• Financial market infrastructures
• Healthcare
• Drinking water
• Wastewater
• Digital infrastructure — IXPs, DNS service providers, TLD name registries, cloud providers, datacenter providers, CDN providers, public electronic communications networks/services
• ICT service management (B2B) — MSPs, MSSPs
• Public administration (central + at member state’s discretion regional)
• Space

Annex II — other critical sectors

• Postal and courier services
• Waste management
• Chemical (manufacture, production, distribution)
• Food production, processing, distribution
• Manufacturing of critical products (medical devices, computers, electronic equipment, machinery, motor vehicles)
• Digital service providers — online marketplaces, search engines, social networking platforms
• Research

Whether an entity is essential or important is decided by Article 3, not by the annex alone. Large entities in Annex I sectors are essential; medium-sized entities in Annex I sectors and entities in Annex II sectors are important, unless a member state designates them otherwise. Some entities are in scope regardless of size (e.g., DNS service providers and TLD registries, which are essential).

Article 21 — risk management measures

The substantive heart of NIS2 cybersecurity requirements. Member states must ensure essential and important entities take appropriate technical, operational, and organisational measures across:

1. Policies on risk analysis and information system security
2. Incident handling
3. Business continuity (backup management, disaster recovery, crisis management)
4. Supply chain security (including security-related aspects in relationships with direct suppliers and service providers)
5. Security in network and information systems acquisition, development, and maintenance (incl. vulnerability handling and disclosure)
6. Policies and procedures to assess effectiveness of cybersecurity risk-management measures
7. Basic cyber hygiene practices and cybersecurity training
8. Policies and procedures regarding the use of cryptography and, where appropriate, encryption
9. Human resources security, access control policies, asset management
10. Use of multi-factor authentication / continuous authentication, secured voice/video/text comms, secured emergency comms

For cloud architecture, that maps to specific controls.

Article 21 → architecture mapping

Risk analysis and information system security

• Risk register documented per critical workload
• Threat model annual review
• Network segmentation (zero-trust by default)
• Pod Security Standards / Kubernetes Network Policies enforced

Incident handling

• 24×7 detection capability for critical workloads
• Documented incident response with named roles
• Blameless post-mortems with action items

Business continuity

• Backup with tested recovery (Velero + per-app PITR for databases)
• DR site in separate region or jurisdiction
• BCP exercises at least annually with documented results
• RTO/RPO documented and tested per critical workload

Supply chain security

• ICT third-party inventory complete
• Critical-function dependencies mapped to second hop
• Contractual security clauses in critical-function agreements
• Continuous monitoring of supplier security posture
• Concentration-risk position documented

Acquisition, development, maintenance security

• SAST / DAST in CI
• Container image scanning + SBOM
• Vulnerability handling SLA defined and met
• Disclosure policy published

Effectiveness assessment

• Annual external assessment recommended for essential entities
• Internal review quarterly
• Metrics tracked over time

Cyber hygiene and training

• Mandatory cybersecurity training for all staff
• Phishing simulation
• Privileged-user training
• Management body cybersecurity training (Article 20)

Cryptography

• TLS 1.2+ for transit
• Encryption at rest for sensitive data classes
• Customer-controlled keys for the most sensitive workloads
• Key rotation and emergency access documented

Access control / human resources

• Workload identity (SPIFFE/SPIRE or equivalent)
• Privileged access management
• Joiner-mover-leaver process automated where possible
• Asset register complete and current

MFA / secured comms

• MFA for all privileged accounts
• MFA for end-user access where reasonable
• Secured comms for incident response

Article 23 — incident reporting

NIS2 mandates a three-stage incident reporting process for significant incidents:

• Early warning within 24 hours of becoming aware
• Incident notification within 72 hours, including initial assessment of severity, impact, and (where applicable) cross-border implications
• Final report within one month, with detailed description, type of threat or root cause, applied/ongoing mitigation, and (where applicable) cross-border impact

For cloud architecture this requires:

• Detection telemetry capable of recognizing significant incidents at a 24-hour timeline (often the bottleneck — alert fatigue masks signal)
• Incident classification process (what counts as significant?)
• Incident response procedures with clear escalation triggers
• Communication channels with national CSIRT / competent authority pre-established
• Documentation discipline so the 72-hour and one-month reports are evidence-based

Article 3 and Article 27 — registration

Many essential and important entities must register with their national competent authority. The architecture decision: ensure your registration data (including DNS, IP ranges, contact details) reflects what’s actually deployed.

Article 12 — coordinated vulnerability disclosure

Member states must designate a CSIRT to coordinate vulnerability disclosures. Entities should publish a vulnerability disclosure policy. Architecture: a structured CSAF feed or Hugo-equivalent for security advisories, with named contact points.

A working NIS2 cloud architecture checklist

Use this during architecture review or production-readiness assessment.

Risk management

• Risk register documented per critical workload
• Annual risk review completed
• ICT asset register complete and current
• Threat model documented
• Cybersecurity policy approved by management body

Incident handling

• 24×7 detection capability for critical workloads
• Documented incident response with named roles
• Incident classification process documented
• Pre-established communication channels with CSIRT/authority
• Reporting templates aligned to 24/72-hour/one-month structure
• Annual incident-response exercise

Business continuity

• BCP plan documented per critical workload
• RTO/RPO documented per critical workload
• Backup tested with recovery in past 12 months
• DR site operational and tested
• BCP exercise within past 12 months

Supply chain

• ICT third-party inventory complete
• Critical-function dependencies mapped to second hop
• Contractual security clauses in critical-function agreements
• Continuous supplier monitoring documented
• Concentration-risk position documented

Cybersecurity hygiene

• MFA for privileged accounts (universal)
• MFA for end-user access where reasonable
• Cybersecurity training mandatory for all staff
• Management body trained
• Privileged-access management deployed
• Joiner-mover-leaver automated

Vulnerability management

• CVE response SLA documented
• SAST/DAST in CI pipelines
• Container scanning + SBOM
• Disclosure policy published
• Coordinated vulnerability response process

Cryptography

• TLS 1.2+ for transit
• Encryption at rest for sensitive data
• Customer-controlled keys where applicable
• Key rotation documented

Audit and effectiveness

• Annual external assessment (essential entities)
• Quarterly internal review
• Audit log retention meets longest applicable requirement
• Metrics tracked over time
• Management body kept informed

Common architectural failures

Failure 1: detection telemetry tuned for performance, not security

Observability built for SLO compliance; not tuned to alert on security-significant events. NIS2 requires the latter.

Failure 2: BCP plan never tested

NIS2 requires BCP. Most plans have never been exercised under realistic failure.

Failure 3: supply chain visible only to first hop

Article 21(2)(d) requires supply-chain security including direct suppliers and service providers. Most entities cannot enumerate beyond first hop.

Failure 4: incident-reporting process undocumented

24/72-hour/one-month timeline is tight. Without pre-established process and templates, the first real incident becomes a scramble.

Failure 5: management body untrained

Article 20 requires management body to follow cybersecurity training. Many compliance programs treat NIS2 as IT-only.

How to assess where you stand

A structured NIS2 readiness assessment is the cheapest insurance before regulator dialog. Ænix runs this as part of Platform Readiness Assessment with sovereignty/regulator workstream emphasized.

For details see the NIS2 compliance services page.

Want to dig deeper?

• NIS2 compliance services — engagement details
• DORA compliance — financial-services regulator
• Data sovereignty — adjacent trigger
• Cozystack — sovereign-by-architecture platform

Why MSPs need cloud now

Enterprise customers increasingly expect cloud capabilities from their MSPs. The traditional MSP offering (managed Microsoft 365, infrastructure management, security operations) is necessary but no longer sufficient. Cloud as part of the offering — branded with the MSP, run on shared or dedicated infrastructure — is what compounds value.

Architecture pattern

• Multi-tier Tenant CRD — Ænix → MSP → MSP customers
• Per-tier isolation — RBAC, quotas, observability scope, billing
• Branded customer-facing portal — Cozystack Dashboard customized per MSP
• WHMCS-integrated billing — flows through MSP’s customer-management
• Service catalog — MSP can curate (e.g., expose only PostgreSQL, hide Kafka if not supported)

Reseller economics

For mid-size MSP (50-500 customers):

• Platform cost — Ænix engagement + hardware + ops
• Per-customer cost — incremental hardware/storage/bandwidth
• MSP customer pricing — typically 30-50% above raw platform cost
• Margin — covers MSP support, sales, operations

Break even at 30-50 paying customers if the platform and its tooling are what you are covering. Budget 50-100 if you are also funding a dedicated on-call rota from day one — that is the same business with a different cost base, not a different answer.

Engagement sequencing

1. Discovery — MSP customer base, service catalog priorities
2. Cozystack pilot — internal validation
3. Initial customer cohort — 5-10 customers with full white-label experience
4. Operations workflow — support escalation, SLA management
5. Scale — open to broader MSP customer base

Total elapsed: 6-12 months.

Regional and specialty cloud is having a moment in 2026. Hyperscaler economics, sovereignty pressure, and post-Broadcom market dynamics have all opened space for non-hyperscaler cloud products that didn’t make sense to launch 5 years ago. Sovereign cloud products from regional providers in the EU, Central Asia and MENA are visible examples. Many more are in stealth or early stages.

Why now

Three independent dynamics support new cloud product launches:

Hyperscaler economics breakdown for some workloads. Sustained inference, high-egress workloads, and regulated workloads are increasingly expensive on hyperscalers relative to dedicated infrastructure. A regional cloud serving these workloads has unit-economic advantages.

Sovereignty as competitive advantage. EU-member-state, Kazakhstan, several APAC jurisdictions have explicit sovereign-cloud mandates. Regional providers serving these mandates have regulatory moat that hyperscalers cannot access.

Post-Broadcom virtualization market disruption. VMware-exit-driven workloads have to land somewhere; regional providers can absorb a portion if they ship the right product.

Six layers of a cloud product

A working customer-facing cloud product has six layers, all of which need engineering:

1. Hardware

Compute servers, storage, network fabric, datacenter / colocation. Sized for initial customer cohort plus growth headroom.

2. Platform

Multi-tenant Kubernetes-native platform with KubeVirt for VMs, Cilium and Kube-OVN for networking, LINSTOR (DRBD-replicated block) for storage. Cozystack is the open-source default for this pattern.

3. Service catalog

What customers can self-provision: VMs, K8s clusters, managed databases (PostgreSQL, MariaDB, MongoDB, Redis, Valkey, Kafka, ClickHouse, OpenSearch, etc.), S3 buckets, GPU instances, networking primitives.

4. Customer-facing portal

Self-service UI (Cozystack Dashboard or custom). Catalog browsing, provisioning, monitoring, billing visibility.

5. Billing

WHMCS production-ready integration in two modes — shipped by Ænix as the WHMCS integration rather than as part of open-source Cozystack. Custom billing for specific markets.

6. Operations

24×7 NOC, customer support, SLA management, observability per tenant, incident response.

Architectural decisions specific to public cloud products

A public cloud product differs architecturally from an internal platform in several ways:

Multi-tenant isolation must be hard. Customers don’t trust each other; regulatory audits are real. Tenant CRD pattern with strong isolation is necessary, not optional.

Self-service must be polished. Internal platforms can have “ask the platform team” as escape hatch. Customer-facing products can’t.

Billing must be accurate from day one. Customers will dispute the first bill; the system has to support that conversation.

Observability for customers, not just for ops. Customers want their own metrics, not just SLA data.

Compliance posture is the product. Sovereignty, data residency, audit-readiness are differentiation features, not afterthoughts.

Go-to-market sequencing

Launch in cohorts:

1. Beta with 3-5 friendly customers — fix the rough edges before paying customers see them.
2. Limited GA with 10-50 customers — learn billing and support patterns at small scale.
3. General availability — open to general market.
4. Specialty expansion — add specific services (more GPU classes, AI services, etc.) based on observed demand.

Timing: at provider scale the platform goes live in weeks once hardware is ready, using the productized installer; the beta and limited-GA cohorts then run at your commercial pace. Multi-region national or operator programmes take longer: a 3-6 month pilot, then 9-18 months to full multi-region operation.

Where launches stumble

Stumble 1: under-engineered multi-tenancy

“We’ll add proper isolation in v2.” Customer #1 finds the gap; reputation damage is hard to recover from.

Stumble 2: billing as afterthought

Billing integration in the last month before launch. Doesn’t work; customers don’t pay; revenue collection is a multi-quarter project.

Stumble 3: operational under-investment

Platform built; ops team sized for 50 customers; they sign 200 in first quarter. Service quality collapses.

Stumble 4: copying hyperscaler architecture wholesale

Designed at hyperscaler scale; over-engineered for actual customer count. Operational complexity exceeds revenue.

Stumble 5: undifferentiated commodity offering

Generic cloud product with no differentiation from hyperscaler. Customers default to AWS / Azure / GCP. Specialty / sovereignty / regional matters as differentiator.

Ænix engagement

Ænix has built customer-facing cloud products end-to-end on Cozystack for hosting providers and regional cloud operators. The engagement structure:

• Free 30-minute discovery call, then a fixed-price Platform Readiness Assessment (14 days focused or 28 days full)
• Platform build — platform live in weeks once hardware is ready; portal, billing, operations workflows and first-cohort onboarding follow. Multi-region programmes: 3-6 month pilot, then 9-18 months to full multi-region
• Ongoing (optional) — support subscription from the published tiers, plus managed services during ramp

For details see the public cloud builder service and Ænix Public Cloud Platform.

What Industry 4.0 actually means in 2026

The term has accumulated marketing weight. Practically, Industry 4.0 manufacturing means:

• IoT instrumentation across production floors
• Real-time data collection from machinery
• AI-driven quality control and predictive maintenance
• Digital twins of production lines
• Supply-chain integration across systems
• OT/IT convergence

All of this requires infrastructure — typically a mix of edge compute (close to machinery), regional/HQ cloud (for analytics, ML, and integration), and integration with existing enterprise systems.

Architectural pattern

HQ Cloud (Cozystack)
   ├── Analytics platform
   ├── ML training
   ├── Enterprise integration
   └── Data warehouse
        ↓ (data flow)
Regional sites (Cozystack)
   ├── Regional aggregation
   ├── Production planning
   └── Quality systems
        ↓
Production-floor edge (Cozystack)
   ├── Real-time control
   ├── IoT data ingestion
   ├── Local AI inference
   └── OT/IT interface

Cozystack runs at all three layers with consistent operational model.

Sovereignty for industrial IP

Industrial IP — design data, formulations, process specifications — has higher sovereignty requirements than typical enterprise data. The architectural answer is an air-gap-capable platform with opt-in volume encryption at rest (LINSTOR and LUKS) and a passphrase the manufacturer holds; the key-management process is designed with you.

NIS2 compliance

Manufacturing of critical products (medical devices, computers, electronic equipment, machinery, motor vehicles) is in NIS2 scope. Architectural implications same as broader NIS2 — see NIS2 compliance.

Most Kubernetes-setup tutorials get you a cluster. They don’t get you a production-ready cluster. The difference is the dozen architectural decisions that don’t show up in a “kubectl apply” workflow but that determine whether your cluster operates well at scale or becomes a permanent maintenance burden.

Decision 1: distribution choice

The question “which Kubernetes distribution” looks like a tooling decision. It’s actually an architectural decision because it determines:

• Operational model (managed vs self-managed control plane)
• Multi-tenancy approach
• Storage and networking integration
• Vendor relationship (open-source vs commercial)
• Long-term cost trajectory

Common 2026 options:

• Vanilla Kubernetes (kubeadm / Cluster API) — maximum control, maximum operational responsibility. Right for organizations with strong platform-engineering capability.
• Cozystack — open-source, CNCF Project, multi-tenant + virtualization (KubeVirt). Right for service providers, sovereign-cloud builders, regulated multi-tenant.
• OpenShift / OKD — Red Hat commercial / community. Right for enterprises already on Red Hat with established procurement relationship.
• Rancher / RKE2 — SUSE-backed, multi-cluster management. Right for distributed cluster fleets.
• Talos — minimal Linux + Kubernetes only. Right as the OS layer underneath Cozystack or vanilla Kubernetes.
• Hyperscaler-managed (EKS / AKS / GKE) — control plane managed; you operate workloads. Right when sovereignty isn’t a constraint and operational simplicity beats cost.

Distribution choice is structural; getting it wrong is expensive to undo.

Decision 2: multi-tenancy

Kubernetes ships with namespaces, not multi-tenancy. The question is how to bridge the gap.

Patterns:

• Soft multi-tenancy (namespaces + RBAC) — single cluster, multiple namespaces, RBAC to separate. Simplest, least isolation. Acceptable for trusting tenants (BUs of one organization).
• Hard multi-tenancy (Tenant CRD) — Kubernetes-native tenant abstraction, with namespace, quotas, RBAC, observability scope, audit trail. Right for service-provider model or regulated multi-tenancy. Cozystack ships this out of the box.
• Cluster per tenant — physical isolation. Highest assurance, highest operational cost. Right for customers requiring full isolation (high-stakes regulated, classified).

The choice depends on the trust model and isolation requirements. Most multi-tenant deployments in 2026 use Tenant CRD pattern — strong isolation without per-tenant cluster overhead.

Decision 3: networking (CNI)

CNI choice in 2026 has converged on:

• Cilium — eBPF-based, network policies + observability + service mesh in one. Default for new deployments. CNCF Graduated.
• Calico — long-standing, BGP-friendly, strong on network policies.
• Flannel — simpler, less feature-rich.
• Cloud-provider CNIs — AWS VPC CNI, Azure CNI, GCP — when running in cloud.

For production multi-tenant deployments, Cilium is increasingly the default — its network policies integrate with the multi-tenancy model and observability is native.

Decision 4: storage

Storage in Kubernetes is a separate operational discipline. Options:

• LINSTOR (DRBD) — replicated block storage. Production-grade for stateful workloads. Cozystack default.
• Rook + Ceph — object + block + file storage. Heavier operational footprint; powerful when justified.
• Longhorn — Rancher-led replicated block storage. Easier than Ceph, less mature than LINSTOR.
• Vendor SAN / hyperconverged — VMware vSAN, NetApp, Pure Storage. Operational handoff to vendor; cost ceiling.
• Cloud-provider storage — EBS, Azure Disks, GCP PD — when cloud-managed.

For production stateful workloads on bare metal, LINSTOR/DRBD are the realistic choices. Cozystack ships LINSTOR; it’s been validated in production.

Decision 5: identity and secrets

Workload identity in Kubernetes:

• SPIFFE/SPIRE — open-source, workload identity standard. Increasingly default for service-to-service auth.
• Cloud-provider workload identity — AWS IAM Roles for Service Accounts, Azure AD Workload Identity, GCP Workload Identity. Locks identity to the cloud provider.
• OIDC + service accounts — Kubernetes-native, federated with workforce identity (Keycloak, Okta).

Secrets management:

• External Secrets Operator + cloud-provider secret stores (AWS Secrets Manager, Azure Key Vault).
• HashiCorp Vault — for organizations already on Vault.
• Sealed Secrets — for GitOps-friendly encrypted-at-rest secrets in Git.

The right combination depends on your existing identity infrastructure.

Decision 6: GitOps engine

Two production-grade options:

• Argo CD — UI-rich, plugin ecosystem, multi-tenancy via projects.
• Flux — closer to upstream Kubernetes, Helm-controller-native, lighter operational footprint.

Both are CNCF Graduated. Cozystack uses Flux. The choice depends on team familiarity and operational style preferences.

Decision 7: observability stack

Three serious options for self-hosted:

• VictoriaMetrics + VictoriaLogs — low-overhead, single-binary, performant at scale. Cozystack default.
• Prometheus + Loki — Grafana ecosystem standard. More memory-intensive; complex at scale.
• Commercial (Datadog, New Relic, Splunk) — SaaS, sovereignty implications.

For sovereignty-sensitive or cost-sensitive deployments, VictoriaMetrics + VictoriaLogs has been our default recommendation since ~2024.

Decision 8: backup and DR

• Velero — Kubernetes-native backup, restore to any cluster. CNCF Project.
• Stash / Kanister — backup operators with stronger app-aware features.
• Cloud-provider backup — managed backup services on hyperscaler clusters.

For DR specifically: per-app patterns (PostgreSQL PITR, etc.) on top of Velero infrastructure is the production pattern.

Decision 9: ingress and load balancing

• Cloud LB + ingress controller (NGINX / Traefik / Contour) — when running in cloud.
• MetalLB + ingress controller — bare metal layer-2 load balancing.
• Cilium L7 load balancing — when Cilium is the CNI.
• Service mesh (Istio / Linkerd) — for advanced traffic management.

Bare-metal deployments default to MetalLB + Cilium today.

Decision 10: lifecycle management

How clusters get upgraded, scaled, and recovered:

• Cluster API — Kubernetes-native cluster lifecycle. Used by Cozystack.
• Kubeadm + manual upgrades — works for small fleet.
• Vendor-managed lifecycle — OpenShift, Rancher, vendor-led.
• Hyperscaler-managed — control plane upgrades automatic.

For production fleets >5 clusters, Cluster API or vendor-managed lifecycle pay back the operational cost.

Operational practices that matter

Beyond architecture, the practices that determine whether a cluster operates well:

SLOs and error budgets

Define cluster-level SLOs (control plane availability, ingress availability, etc.). Define workload-level SLOs per service. Use error budgets to drive prioritization.

Documented runbooks

Common operational scenarios documented: pod failure, node failure, control-plane recovery, certificate rotation, storage failure, network failure. Practice them.

Capacity planning

Compute, memory, and storage capacity tracked. Forecasted against workload growth. Decisions made before resource exhaustion.

Upgrade discipline

Kubernetes minor versions every quarter; cluster upgrades quarterly or twice-yearly. Skip-version upgrades are not supported by upstream — keep current.

Incident response

On-call rotation, incident commander role, blameless post-mortems with documented action items.

Security posture

Pod Security Standards enforced; network policies as default-deny; secrets in cloud secret stores or Vault; image scanning in CI; runtime threat detection where applicable.

Common production-readiness failures

Failure 1: dev cluster scaled to prod

Cluster started for dev; workloads moved to prod without re-architecting. Falls over the first time prod load arrives.

Failure 2: no observability budget

Prometheus runs out of memory at production scale; metrics retention dropped to 7 days; observability becomes useless. Should have been planned upfront.

Failure 3: security as afterthought

Pod Security Standards not enforced; network policies absent; secrets in plain ConfigMaps. Audit findings drive emergency remediation.

Failure 4: upgrade debt

Kubernetes 3-4 minor versions behind; upgrades are big-bang projects; technical debt compounds.

Failure 5: no platform team

Cluster is “owned” by everyone, operated by no one. Drift accumulates; nobody can confidently change anything.

How to assess what you have

Before building or scaling, an architecture review is the cheapest insurance. The output is a written assessment of where you stand, where the gaps are, and what production-readiness looks like for your scale.

Ænix runs Kubernetes architecture reviews as part of the fixed-price Platform Readiness Assessment: 14 days for a focused review, 28 days for the full assessment.

Want to dig deeper?

• Kubernetes consulting services — engagement details
• Platform engineering services — broader scope
• Internal developer platform — IDP layer
• Cozystack — open-source platform foundation

Most “should we build our own cloud product?” conversations at hosting providers stop at the technology question. The harder question is the unit economics: what does it cost per tenant, what’s a realistic ARPU, how many tenants until break-even, and where does the model fail.

This article is the working version of that conversation. It assumes the technology decision is settled (the Cozystack-based Ænix Public Cloud Platform) and focuses on whether the economics fit your hosting business — not the abstract one.

What Public Cloud Platform actually delivers

Before economics, scope. Public Cloud Platform is a complete public-cloud product Ænix sells to hosting providers, MSPs, regional clouds, and small-to- mid data centres. It includes:

• Multi-tenant Cozystack platform running on customer-controlled bare metal (KubeVirt + Cilium + Kube-OVN + LINSTOR + Tenant CRD).
• Cozystack Dashboard — customer-facing self-service portal, brandable to your hosting brand.
• WHMCS integration — billing flows through the customer-management system most hosting providers already operate.
• Service catalog — VMs, tenant Kubernetes clusters, managed databases (PostgreSQL, MariaDB, MongoDB, Redis, Valkey, Kafka, ClickHouse, etc.), S3-compatible object storage, GPU services. Curatable per provider.
• Tenant lock / suspension — operational hooks for non-payment and policy enforcement.
• Migration tooling — productized patterns for VMware, OpenStack, Virtuozzo, Proxmox sources.

What it is not: a hyperscaler. It is a sovereign, multi-tenant cloud product for hosting providers who want to compete on regional presence, sovereignty, and pricing flexibility — not on hyperscaler-scale catalog depth.

Pricing model

Public Cloud Platform subscriptions use the published support tiers, priced per 10 physical nodes per month on annual billing: Basic $1,250, Standard $3,000, Plus $5,500; Enterprise is quoted individually. Higher tiers add faster response times, unlimited incidents, 24×7 support (Plus and Enterprise) and a wider support scope — the full matrix is on /pricing/. Ænix does not charge per VM, per CPU, or per GB — the Cozystack platform itself is free under Apache 2.0; what you pay for is engagement, support, and operational assurance. Every support tier includes the proprietary Ænix commercial modules (billing system and WHMCS integration).

For a typical mid-size hosting provider running 30-100 customer-facing nodes, that is $3,750-12,500/month on Basic or $9,000-30,000/month on Standard at list price. Compare it with the recurring licence and subscription cost you pay VMware or an OpenStack distribution vendor today.

Unit economics — per-tenant view

The economics question every hosting CFO asks: what does it cost us to serve one tenant, and what can we realistically charge?

Cost per tenant

Infrastructure cost per tenant is dominated by the underlying compute, storage, and bandwidth, not by Cozystack itself. Cozystack overhead is ~5-10% of node capacity (typical Kubernetes-platform overhead, well within acceptable for production). For a tenant consuming roughly:

• 2 vCPU
• 4 GB RAM
• 50 GB block storage (replicated 3×)
• 100 GB egress / month

Direct infrastructure cost (amortised hardware + colocation + bandwidth) in 2026 European pricing is typically €15-30/month. Cozystack platform overhead allocation (the platform team’s salary spread across all tenants) on a mid-size provider running 500 tenants adds another €5-10.

So all-in cost per typical tenant: €20-40/month at the lower end of the resource consumption profile.

What you can charge

Hosting-provider ARPU for a comparable resource profile in 2026 EU markets typically lands in €40-80/month. Higher in DACH and Western Europe, lower in Central / Eastern Europe and Central Asia. Add managed services (managed PostgreSQL, managed S3, GPU access) and ARPU lifts to €80-200+/month per tenant.

That puts margin at roughly 2-3× cost at the low end, 4-6× cost on managed-service-heavy tenants. Not hyperscaler-margins; not VMware reseller margins either. Closer to traditional hosting margins in the post-Broadcom 2026 reality.

Break-even math

The other CFO question: how many customers until we make money?

The fixed cost stack for a mid-size hosting provider on Public Cloud Platform:

Total monthly fixed: €44-85k, plus $15k for Ænix support.

With €25-50/month margin per tenant (€40-80 ARPU after €15-30 direct infrastructure cost), break-even sits at roughly 1,200-4,000 paying tenants depending on ARPU mix and where you are in the salary band. This is the full-programme case with a dedicated team; a 10-node start on the Basic tier with existing staff breaks even far earlier — model your own numbers in the ISP calculator.

For providers currently running ~500 customers on legacy infrastructure who are evaluating the move, this matters: you need a credible path to at least double tenant count within 18-24 months for the economics to actually work. Without growth, Public Cloud Platform is a cost reduction (modest) but not a transformation.

For providers below ~300 customers, a full programme with a dedicated platform team is often premature — that fixed cost overwhelms the margin contribution. A smaller start (10 nodes on the Basic tier, a narrow catalogue, existing staff) is usually the better first step. We’ll say so in a discovery call rather than push a larger engagement.

Where the model breaks

Three failure patterns recur:

1. Customer-facing portal under-investment

Hosting providers historically compete on price and reliability. The Cozystack Dashboard out of the box is functional but generic; differentiation comes from polish (UX flows that match how your customers think about ordering, configuring, paying). Providers who treat the portal as “good enough” lose conversion to providers who invest in it.

Ænix engagement includes Cozystack Dashboard brand customization; deeper UX work is typically a separate Phase 2.

2. Service-catalog mismatch

Cozystack offers 20+ managed services; not all of them fit every provider’s customer base. Exposing all of them without operational backing means customers ordering Kafka or ClickHouse and discovering the provider can’t really support them. Curate the catalog to what you can actually operate at the SLA you promise. Service rollout in cohorts is the standard playbook.

3. Operations team under-staffed for growth

The biggest single failure mode in our pipeline: Public Cloud Platform deployed, launches successfully, signs 200 customers in the first quarter — and then the 4-person operations team that worked at 50 customers can’t scale. Customer support response time degrades, SLA breaches multiply, churn picks up.

Plan operations team size for 18-month-out customer count, not current. Hire ahead.

How Public Cloud Platform compares to alternatives for hosting providers

Versus VMware Cloud Director (vCD):

vCD is the historical incumbent for hosting providers. Post-Broadcom, subscription pricing has reshaped the math — 2-5× increases on renewal, mandatory VCF bundling, end of perpetual licensing. For most providers running vCD today, the renewal cycle is the trigger. Ænix Public Cloud Platform goes live in weeks once hardware is ready; moving an existing vCD estate onto it is a separate migration project, sized by estate in the assessment.

Versus OpenStack:

OpenStack remains valid for providers with deep OpenStack expertise and large-scale (>500 nodes) deployments where operational complexity is amortised. For mid-size providers, OpenStack’s operational footprint (50+ services, distinct upgrade lifecycles per component) overshoots what the team can sustain. Public Cloud Platform substantially smaller surface.

Versus building it yourself on vanilla Kubernetes + KubeVirt + Helm:

This is the credible alternative for providers with strong platform engineering capacity. Trade-off: 12-24 months of build time + ongoing maintenance versus a turnkey deployment. We’ve seen both work; the build-it-yourself path is the right choice when you have a 10+ engineer platform team and the components match your specific operational preferences. For the typical mid-size provider with a 3-5 engineer platform team, Public Cloud Platform wins on time-to-market and operational predictability.

Versus a hyperscaler-managed cloud product (white-label):

Hyperscalers (AWS, Azure, GCP) sometimes offer hosting partners white-label or co-branded cloud arrangements. Trade-off: lower operational complexity for the provider, but per-customer margin is typically lower, and sovereignty positioning is weaker (the provider still depends on the hyperscaler, which European customers increasingly view as a structural risk).

When Public Cloud Platform is the right answer

It fits when at least three of the following hold:

1. You operate today on bare metal or commercial hypervisor with recurring licence pressure — VMware, OpenStack, Virtuozzo, or commercial KVM distribution.
2. You have direct customer relationships you can monetise — not pure reseller of someone else’s cloud.
3. You have or can hire a 3-5 engineer platform team — Cozystack needs operational ownership.
4. You target regional, regulated, or sovereignty-sensitive customers — where European-EU positioning matters.
5. Your customer count is 300+ today or you have credible growth path to 1,000+ — for fixed-cost amortisation.
6. You’re willing to invest in customer-facing portal polish — not just treat Cozystack Dashboard as “good enough.”

Fewer than three: usually a different answer is better — staying on existing infrastructure with cost optimisation, partnering with a larger sovereign provider as their channel, or going hyperscaler- managed-cloud-product as a smaller-margin route.

Engagement structure

For providers where Public Cloud Platform fits:

• Discovery call (30 min, free)
• Platform Readiness Assessment (fixed price, 14 days focused or 28 days full) — current estate inventory, target architecture, migration plan
• Platform live and pilot cohort — the platform goes live in weeks on your hardware with the productized installer; 5-10 friendly customers migrated, billing validated
• Limited GA (2-4 months) — 50-100 customers, operational workflows stabilised
• General availability — open market launch
• Support subscription (ongoing) — one of the published tiers; Plus or Enterprise for 24×7 coverage

How long the commercial launch takes after the platform is live depends on migration scope and team readiness. Multi-region national or operator programmes follow a 3-6 month pilot, then 9-18 months to full multi-region operation.

Where to dig deeper

• Public Cloud Platform landing page — feature list, pricing block, FAQ
• Hosting providers industry page — hosting-provider-specific positioning
• White-label cloud services — for MSP / channel-partner extensions of the model

Why K-12 is different from universities

Universities have research computing, AI/ML labs, and curriculum-driven cloud-native education needs. K-12 districts:

• Don’t run research computing
• Don’t teach Kubernetes (mostly)
• Have student-data privacy as primary infrastructure concern
• Operate at consumer-facing scale (10K-100K+ students)
• Have long budget cycles (3-5 year procurement)

These different drivers mean different architectural answers.

When K-12 needs sovereign infrastructure

Three scenarios:

1. Regulatory pressure — some EU member states require student-data residency under national privacy regulations
2. Liability / public concern — high-profile districts that publicly committed to local data residency
3. EdTech platform development — districts building their own learning platforms

Most K-12 districts fall in none of these — hyperscaler-managed services + standard EdTech tools is right.

Architecture pattern for fitting K-12

• District-tier cluster — Cozystack at central district IT
• Per-school isolation — Tenant CRD per school
• Sovereign by architecture — student data on customer hardware, opt-in volume encryption with a passphrase the district holds
• Standard EdTech integrations — Google Classroom / Microsoft 365 federation

For consortia (multi-district shared platform):

• Federated multi-district platform
• Shared core, per-district isolation
• Joint procurement, distributed operations

Common pitfalls in K-12 platform projects

• Underestimating EdTech vendor integration complexity
• Skipping FERPA / GDPR audit-readiness
• Vendor-led “education cloud” with lock-in
• Mid-cycle re-architecture due to budget cycle mismatch

The “IDP” acronym is overloaded. It means both:

• Internal developer platform — the underlying capability stack (Kubernetes, IaC, observability, golden paths).
• Internal developer portal — the UI/catalog layer (Backstage, Port, custom).

Most discussions confuse these. The actual question — “do we need Backstage?” — has different answers depending on which IDP you’re talking about.

Platform vs portal

A platform without a portal still works. A portal without a platform is wallpaper.

Platform answers: “What capabilities can product teams self-serve?”

• Environment provisioning
• Application deployment
• Database / queue / cache provisioning
• Observability onboarding
• Secrets, identity, network access

Portal answers: “How do product teams discover and access those capabilities?”

• Service catalog
• Documentation entry point
• Self-service forms / actions
• Cost / SLO dashboards per service

The platform is required for self-service to work. The portal is required for discoverability when service count grows past what teams can hold in their heads.

When you need a portal

A portal becomes valuable when:

• Service count is large — 50+ services where engineers can’t navigate all of them.
• Team size is large — many engineers, many of whom aren’t yet platform-fluent.
• Cross-team service discovery is real — engineers from team A consume services from team B.
• Compliance / audit requires service catalog — regulator-ready service inventory.

When none of these hold (small org, small service count, fluent platform team), a portal adds maintenance cost without much benefit.

Portal options compared

Backstage (CNCF Incubating)

What: Open-source service catalog + plugin ecosystem. Originally built by Spotify, now CNCF.

Strengths: Mature, broad plugin ecosystem, customizable, strong community.

Weaknesses: Operational cost is real — running Backstage means maintaining plugins, integrating with internal tools, evolving with Backstage’s release cadence. Underestimated by many adopters.

Best for: Mid-to-large engineering orgs (200+ engineers) with platform-team capacity to maintain Backstage as a product.

Port (port.io)

What: SaaS internal developer portal.

Strengths: No self-hosting; quicker to deploy than Backstage; opinionated.

Weaknesses: SaaS dependency (sovereignty implications); less customizable than Backstage.

Best for: Mid-size orgs willing to use SaaS, wanting to skip Backstage’s operational cost.

Cortex / Compass / custom variants

Adjacent options with different trade-offs. Cortex is engineering-effectiveness-focused; Compass is Atlassian-native; custom is “build it yourself.”

No portal

What: IaC repository + Markdown documentation + GitOps interface.

Strengths: Zero operational cost beyond Git. No portal-specific maintenance.

Weaknesses: Discoverability scales poorly past 30-50 services.

Best for: Smaller orgs (under ~100 engineers) where the portal cost would exceed its value.

Backstage’s actual best fit

Backstage works well when:

• 200+ engineers, 100+ services
• Platform team has 5+ engineers who can dedicate time to maintenance
• Plugin ecosystem matches your tooling
• Customization is a priority (you’ll write or extend plugins)

When those don’t hold — the operational cost overshoots the value, and a lighter-weight option fits better.

CNOE — the open-source platform-engineering reference

The CNOE (Cloud Native Operational Excellence) industry initiative is worth mentioning. It’s an opinionated reference architecture combining Backstage, Argo CD, Crossplane, External Secrets, and other CNCF tools into a coherent platform pattern. For organizations that want a “platform-in-a-box” using CNCF projects, CNOE is the structured starting point.

CNOE is complementary to Cozystack: CNOE focuses on the developer portal + tooling layer; Cozystack focuses on the underlying multi-tenant Kubernetes-native platform with virtualization. Both can coexist.

How to decide

A practical decision tree:

1. Do you have a working platform underneath? If no, build that first. Portal adoption assumes platform exists.
2. Service count over 50? Engineering org over 100? If yes, a portal helps. If no, a portal is over-engineering.
3. Backstage operational cost realistic? If yes (large team, plugin-friendly culture), Backstage. If no, Port or Cortex.
4. SaaS acceptable? If yes, Port or Cortex. If no, Backstage or custom.
5. Will you actually customize? If yes, Backstage. If no, Port.

The decision is smaller than vendors make it sound. The bigger architectural decision is the platform underneath.

Most articles about internal developer platforms in 2026 are still framed as “how to use Backstage.” That framing is wrong. Backstage is a useful tool when the catalog discipline is mature; it is not the platform. The platform sits underneath, and the architecture decisions that matter are made there.

This is what working internal developer platforms actually look like.

The IDP vs IDP problem

The terminology is confusingly overloaded. “IDP” gets used for both:

• Internal developer platform — the capability stack: compute, storage, networking, identity, observability, deployment automation, golden paths.
• Internal developer portal — the UI/catalog layer: Backstage, Port, Cortex, custom dashboards. Sits on top of the platform.

A platform without a portal still works. A portal without a platform is wallpaper.

In practice, organizations get confused, buy a portal, and discover that adoption stalls because the underlying capabilities aren’t actually self-service — they’re just better-documented from the catalog.

This article focuses on platform; portal layer is a smaller, secondary decision.

Six IDP patterns from production

Pattern 1: Multi-tenant Kubernetes-native cloud platform

What: Single Kubernetes cluster (or fleet) with hard-multi-tenancy. Tenant CRD, per-tenant quotas, RBAC, observability scope. KubeVirt for VM workloads, regular containers for everything else, managed databases as Kubernetes-native services.

Used by: Service providers (multi-customer), enterprises with strong BU separation, sovereign-cloud builders.

Example stack: Cozystack (open-source CNCF Project).

Why it works: Single platform serves both VM-heavy and container-heavy workloads. Multi-tenancy is structural, not bolted-on.

Pattern 2: GitOps-first per-team Kubernetes

What: Each product team gets its own Kubernetes namespace (or cluster), provisioned via IaC. GitOps engine (Argo CD or Flux) deploys applications. Platform team maintains the foundation; product teams own their namespace.

Used by: Mid-size organizations (50-500 engineers) with disciplined GitOps culture.

Example stack: Vanilla Kubernetes + Flux + Crossplane + Argo Workflows.

Why it works: Lightweight platform team, high product-team autonomy, clear isolation boundaries.

Pattern 3: Service-template + golden-path platform

What: Pre-built service templates (HTTP API, batch worker, scheduled job) with embedded observability, deployment, and alerting. New service = run a template; everything below is automatic.

Used by: Engineering organizations with high service-creation rate.

Example stack: Cookiecutter or similar templates + Helm charts + Argo CD + auto-instrumented observability.

Why it works: Reduces time-to-first-production-deploy to hours; standardizes operational surface.

Pattern 4: PaaS-lite layered on Kubernetes

What: Heroku-like developer experience built on Kubernetes. git push → CI builds → image pushes → automatic deploy. No Kubernetes manifests in product-team-facing surface.

Used by: Organizations where Kubernetes literacy isn’t universal across product teams.

Example stack: Knative + custom controllers, or Cloud Foundry-style abstractions on Kubernetes.

Why it works: Lowest cognitive overhead for product teams. Trade-off: less flexibility for non-standard cases.

Pattern 5: Backstage-first with capability operators

What: Backstage as the entry point; product teams pick from the catalog. Behind each catalog entry, a Kubernetes operator provisions the requested resource (database, queue, environment). Platform team maintains the operators.

Used by: Organizations with strong product-management discipline at platform-team level.

Example stack: Backstage + Crossplane + custom Kubernetes operators per service type.

Why it works: Catalog-first UX is intuitive for product teams; operators handle the actual provisioning.

Trade-off: Platform team must invest in product-management practices (catalog hygiene, deprecations, roadmap).

Pattern 6: External-services-as-platform

What: Platform team provides catalog of external SaaS services (managed databases, observability, identity) via standardized integration. The “platform” is mostly integration glue + provisioning automation.

Used by: Cloud-native organizations comfortable with hyperscaler-managed services and SaaS dependencies.

Example stack: Crossplane (with cloud providers + SaaS providers) + Backstage catalog.

Why it works: Smallest platform-team footprint; rides hyperscaler reliability.

Trade-off: Vendor lock-in, sovereignty concerns, cost ceiling.

Tools that show up across patterns

The tooling landscape has converged. Across the 6 patterns above:

Compute / orchestration

Kubernetes is the de facto orchestration layer. Distribution choice depends on operational model:

• Cozystack for multi-tenant + virtualization (open-source, CNCF Project)
• OpenShift for enterprise commercial
• Vanilla Kubernetes for simplicity
• Talos as the OS underneath

IaC

Terraform / OpenTofu for cloud and infrastructure. Crossplane for Kubernetes-native infrastructure abstraction. Pulumi as a TypeScript-friendly alternative.

GitOps

Argo CD and Flux are the two production-grade GitOps engines. Both work; Flux is closer to upstream Kubernetes way; Argo CD has stronger UI ergonomics.

Internal developer portal

Backstage dominates. Alternatives: Port, Cortex, Compass, custom. Smaller-scale: a Markdown documentation site backed by a service-catalog YAML in Git.

Observability

VictoriaMetrics + VictoriaLogs (open-source, low-overhead). Prometheus + Loki (still common). Commercial: Datadog, New Relic, Splunk Cloud — but check sovereignty implications.

Secrets and identity

External Secrets Operator + cloud-provider secret stores or HashiCorp Vault. SPIFFE/SPIRE for service identity at scale.

CI/CD

GitHub Actions for source-of-truth; Argo Workflows or Tekton for Kubernetes-native pipelines.

How to pick a pattern

The decision tree:

1. Multi-tenancy needed? (Service-provider model, hard BU separation, regulated multi-tenant.)

   • Yes → Pattern 1 (Multi-tenant Kubernetes-native, Cozystack-style).
   • No → continue.

2. Strong product-team autonomy desired? (Mature engineering culture, GitOps discipline.)

   • Yes → Pattern 2 (GitOps-first per-team).
   • No → continue.

3. High service-creation rate? (>10 new services per month, microservices-heavy.)

   • Yes → Pattern 3 (Service-template + golden path).
   • No → continue.

4. Kubernetes literacy uneven across product teams?

   • Yes → Pattern 4 (PaaS-lite on Kubernetes).
   • No → continue.

5. Strong product-management discipline at platform-team level + appetite for catalog UX?

   • Yes → Pattern 5 (Backstage-first with capability operators).
   • No → Pattern 6 (External-services-as-platform).

This is a heuristic, not a formula. Real engagements often hybrid two patterns.

What about the portal?

Once the platform pattern is chosen, the portal decision is smaller:

• Backstage when service catalog discipline is real, the platform has 50+ catalog entries (services + capabilities), and a platform team can sustain Backstage’s plugin maintenance.
• Port / Cortex / Compass when Backstage seems too much engineering effort and a SaaS portal is acceptable.
• Custom Markdown site + YAML catalog for organizations under 100 engineers; sufficient and cheaper.
• No portal for very small platforms; the IaC repo is the documentation.

A portal is a service-discovery and self-service entry point. It’s not the platform; not the differentiator.

Common architectural mistakes

Mistake 1: starting with the portal

Buying Backstage before the platform exists puts the cart before the horse. Adoption stalls because the catalog points to weeks-long ticket processes.

Mistake 2: too many opinions, too rigid

Golden paths should be golden, not gold-plated. Product teams need escape hatches for non-standard cases.

Mistake 3: under-invested platform-team headcount

Platform team that absorbs both build and on-call duties for shared services becomes ticket support. Build velocity collapses.

Mistake 4: vendor-led platform-in-a-box

Buying a “complete IDP solution” rebuilds the lock-in problem with a different vendor.

Mistake 5: building for engineering elegance, not adoption

Platform team’s customers are product teams. Architecture optimized for engineering elegance often produces a platform nobody adopts.

Want to dig deeper?

• Internal developer platform services — engagement details
• Platform engineering services — broader scope
• Platform engineering vs DevOps vs SRE — terminology
• Cozystack — multi-tenant Kubernetes-native foundation

Hybrid cloud as a term has accumulated marketing weight. Vendors describe almost any non-pure-public-cloud architecture as “hybrid.” The architecturally-relevant question is more specific: what coherent integration pattern connects your private and public substrates, and does it actually work for your workload portfolio?

This is the working version.

What hybrid cloud actually means

Three definitions, in increasing order of architectural usefulness:

1. Hybrid as workload distribution. Some workloads in public cloud, some in private cloud, no integration. Most “hybrid” architectures stop here. (Practical but operationally fragmented.)
2. Hybrid as data flow. Some workloads in each substrate, with explicit data flow architectures connecting them. (Architecturally honest about cross-substrate dependencies.)
3. Hybrid as unified platform. Single platform abstraction running on multiple substrates, with consistent operations, deployment, and observability. (Where the leverage is.)

We engineer toward (3) because (1) becomes operational debt, and (2) is half the work.

Five working hybrid patterns

Pattern 1: Steady-state on-prem + elastic in public cloud

What: Predictable steady-state workloads (databases, batch processing, internal apps) on private cloud. Elastic spike workloads (customer-facing web, ML training) in public cloud. Cross-substrate data flow is designed (not accidental).

Best for: SaaS companies with predictable steady-state customer count + spike-y customer-facing patterns. Most common hybrid pattern in 2026.

Operational pattern: Two substrates with shared platform abstraction (Cozystack on private + Cozystack control plane managing public-cloud Kubernetes). Workloads are portable in principle; in practice, steady-state ones don’t move.

Pattern 2: Critical on-prem + non-critical in public cloud

What: Regulated workloads (banking, healthcare, public-sector) on private cloud. Auxiliary workloads (analytics, internal tooling, dev/test) in public cloud. Sovereignty drives the split.

Best for: Financial services, healthcare, public-sector. The default when DORA / sectoral / data-residency rules drive architecture.

Operational pattern: Stricter isolation between substrates than Pattern 1. Cross-substrate flows audited and minimized. Audit-readiness drives architecture.

Pattern 3: Geographic split

What: EU workloads in EU on-prem; non-EU workloads in regional public cloud. Driven by sovereignty rather than cost or workload type.

Best for: Multinational enterprises with strong sovereignty pressure in some markets but commercial freedom in others.

Operational pattern: Per-jurisdiction tenant isolation. Cross-jurisdiction data flows under explicit legal basis (SCCs, BCRs, etc.).

Pattern 4: Edge + core hybrid

What: Centralized core (private cloud or hyperscaler region) plus edge sites at customer / branch / factory locations. Cross-edge synchronization handled at platform level.

Best for: Telco edge compute, retail, manufacturing, smart-grid operators.

Operational pattern: Edge sites are smaller, simpler instances of the platform; core has full feature surface.

Pattern 5: AI-specific split

What: AI training and inference on dedicated GPU infrastructure (private cloud); rest of business on public cloud or hybrid mix. AI workload economics drive the split.

Best for: AI-heavy organizations where sustained GPU utilization makes hyperscaler economics untenable. Increasingly common in 2026.

Operational pattern: AI cluster operates as a sub-platform; data and model flows between AI cluster and other workloads are designed.

What makes hybrid actually work

Beyond the pattern, three architectural principles separate working hybrid from fragmented multi-cloud:

Principle 1: one platform abstraction

The Kubernetes API (or equivalent) is the lingua franca across substrates. Workload deployment, observability, identity, and operational practices are consistent. The platform team operates one platform, not three.

Cozystack supports this directly: same control plane managing customer-hardware clusters, public-cloud-region deployments, and edge sites.

Principle 2: workload portability where it matters

Critical workloads use platform abstractions that work on multiple substrates. Non-critical workloads can be hyperscaler-native if that’s the right trade-off.

Principle 3: explicit data flow control

Cross-cloud and cross-region data flows are designed, costed, and monitored. Egress is a budget line; sovereignty implications are documented; latency profiles known.

Common failure modes

Failure 1: hybrid as fragmented teams

Public-cloud team and on-prem team operating separately. Different tooling, different deployment patterns, no shared platform. “Hybrid” in name; multi-team chaos in practice.

Failure 2: cloud bursting nobody uses

Architecture supports bursting from on-prem to public cloud for capacity overflow. In production, bursting capability is theoretical — cross-substrate data movement is too slow. Architecture is over-engineered for unused capability.

Failure 3: vendor-led hybrid platform

A vendor sells a “complete hybrid solution” running their software in your datacenter and theirs. Lock-in is structural; vendor’s roadmap becomes yours.

Failure 4: operational drift

Same workload runs differently on public cloud and on-prem because tooling diverges over time. Portability degrades; cost of moving workloads grows.

Failure 5: implicit data flows

Cross-substrate traffic emerges from accidental architecture decisions. Egress costs surprise; sovereignty audit finds compliance gaps.

When hybrid is wrong

A few honest cases where pure architecture beats hybrid:

• All workloads are elastic and not regulated — pure public cloud usually wins.
• All workloads are steady-state, regulated, and modest in scale — pure private cloud usually wins.
• Engineering organization is small — operating one substrate is hard enough; two is harder.
• Workload portability genuinely doesn’t matter — pure hyperscaler-native architecture has its merits.

The honest engagement says “stay pure” when that’s the answer.

Implementation sequence

A practical sequence for moving from fragmented multi-cloud to coherent hybrid:

1. Workload classification — every workload labeled “this substrate, this reason.”
2. Platform foundation — Cozystack (or chosen platform) deployed on private substrate; control plane reach extended to public-cloud regions.
3. Migration cohort 1 — handful of workloads moved to the unified platform pattern; pattern validated.
4. Operational unification — observability, identity, deployment unified across substrates.
5. Migration cohorts 2-N — remaining workloads aligned with the pattern.
6. Steady state — single platform team, single operations model, multiple substrates.

Total elapsed: typically about 8-12 months for ~100 VMs and 18-24 months for ~1,000 VMs, including planning and migration waves.

Want to dig deeper?

• Hybrid cloud platform services — engagement details
• Cloud repatriation — when leaving public cloud
• Private cloud consulting — private side
• Cozystack — the platform

The “should we invest in a platform team?” conversation tends to stall at one of two places: either the CFO can’t see the economic case (“we already pay DevOps engineers, why add more headcount?”), or the engineering organisation tried Backstage-as-a-platform once, found it shallow, and has lost trust in the category.

This article walks through both. What does the cost of developer drag actually look like in numbers? What does an IDP that works deliver? And what does the developer self-service layer of Ænix Private Cloud Platform do that you’d otherwise have to build?

The cost of developer drag

The most consistent finding across our platform-readiness assessments of 200-engineer-plus organisations is that time-to-environment is 2-6 weeks for what should be a 30-minute self-service action.

Where does the time go? Typically:

• IAM: a ticket to the security team, 2-5 days
• Network connectivity: a ticket to network engineering, 3-7 days
• Observability onboarding: ad-hoc, often discovered as missing late
• Compliance review: 1-3 days, sometimes longer for regulated data
• Approval chain: 2-4 days, occasionally longer in matrixed orgs

Each step requires a handoff, which means context loss, which means re-confirmation, which means more time. The standalone steps are small; the cumulative friction is large.

At 5% of engineering productivity (rough estimate from our engagements matching the 200-engineer scale and 2-6 week time-to-environment), the cost of drag on a 200-engineer organisation is ~10 engineers’ worth of lost throughput per year — roughly €1.5-2M at fully loaded rates. Compressing time-to-environment to hours captures most of that.

The internal-developer-platform investment that delivers this compression typically pays back inside 12 months for organisations of this scale.

What “platform that works” actually delivers

For a credible IDP, five characteristics matter more than tool choice:

1. Faster than the ticket alternative

If self-service takes 2 days and a ticket takes 3 days, teams will take the ticket — waiting is easier than learning. The platform has to be substantially faster for adoption to flip.

2. Reliable enough to trust

The self-service path works the first time, every time, for the documented use case. If it breaks 1 in 10 times, teams stop trusting it. Adoption stalls.

3. Documented in one page or less

If the docs are longer than a page, the architecture is too complex. Real golden paths are simple by design.

4. Owned by a real team

A team maintains the path, fields edge cases, ships improvements. Without ownership, paths decay. The platform-engineering function must be a function, not a hobby.

5. Has escape hatches

Product teams can deviate when their case is special. The escape is to a real conversation with the platform team, not “use the path or fail.”

What developer self-service on Private Cloud Platform ships

The developer self-service layer of Ænix Private Cloud Platform is the productisation of these characteristics on top of the Cozystack foundation. Specifically:

A multi-tenant Cozystack platform with Tenant CRD

Every product team gets a Tenant — a Kubernetes-native object with its own namespace, quota, RBAC, observability scope. Per-tenant isolation without per-team cluster overhead. Nested tenants support business-unit hierarchies.

This solves the “soft multi-tenancy is too leaky, cluster-per-team is operationally expensive” trilemma without compromise.

Golden-path-first Cozystack Dashboard

The Cozystack Dashboard in Private Cloud Platform exposes opinionated paths for the 5-10 most common product-team needs: environment provisioning, application deployment, managed-database provisioning, observability onboarding, secrets management. Each path completes in minutes.

Curated, not exhaustive — what your platform team can actually back with support, not every Cozystack capability.

GitOps automation

Argo CD and Argo Workflows pre-integrated with the platform. GitLab integration for the most common enterprise SCM. Product teams commit IaC; the platform reacts. No clicky-clicky for production changes.

Pre-built service templates

Standard service templates (HTTP API, batch worker, scheduled job) with embedded observability, deployment, alerting. New service = template instantiation. Time-to-first-production-deploy is hours, not weeks.

Internal-product-management discipline built in

The platform team is treated as a function with product-team customers. The engagement includes platform-team RACI, internal-NPS metrics, deprecation policy templates, roadmap-management patterns. The discipline is often the missing piece.

What stays the customer’s responsibility

Developer self-service on Private Cloud Platform is the platform substrate plus the operational discipline. Several things remain yours:

• Defining your specific golden paths — the 5-10 you build first depend on what your product teams actually request most.
• Platform-engineering team sizing — typical mature ratio is 1 platform engineer per 10-20 product engineers; we recommend hiring ahead of demand.
• Adoption — a great platform that nobody uses is a sunk cost. Platform-team product-management practices (interviewing product teams, measuring adoption per path, sunsetting paths that don’t get used) need to happen.

Ænix’s engagement model supports all three but doesn’t replace customer ownership. Platforms that customers don’t own organisationally don’t outlast the engagement.

The economic case versus alternatives

Versus DevOps-only

DevOps without separate platform engineering scales linearly with team count — every product team figures out infrastructure, observability, identity, release engineering independently. Above ~50 engineers the duplication overwhelms the savings. Above ~200 engineers it’s an operational drag.

Developer self-service on Private Cloud Platform shifts the model: platform engineering compounds sublinearly with team count. Adding the 21st product team doesn’t add 21st-team’s-worth of platform overhead — the existing platform absorbs them.

Versus building it yourself on vanilla Kubernetes + Backstage

This is a credible alternative for organisations with strong platform engineering capacity and clear architectural opinions. Trade-off: 12-24 months of build time before the platform reaches “production ready” for product teams; ongoing platform-component maintenance overhead.

Ænix Private Cloud Platform, which includes developer self-service, delivers the platform substrate within a 3-12 month build, with ongoing Ænix support. For organisations not staffed for a 12-24 month build, this is the difference between platform engineering happening this year or in 2028.

Versus Backstage-as-platform anti-pattern

Backstage is a portal, not a platform. Buying Backstage before the underlying capabilities are self-service produces a beautiful catalog over the same operational chaos. Adoption stalls.

Private Cloud Platform’s Cozystack Dashboard can be replaced or augmented by Backstage if the customer prefers — but the underlying capabilities (environment provisioning, observability, secrets, identity) are self-service because the platform is, not because the portal pretends they are.

Versus hyperscaler-managed Kubernetes (EKS / AKS / GKE)

Hyperscaler managed Kubernetes is operationally simpler than self- managed clusters. Trade-off: vendor lock-in (control plane is the vendor’s; you can’t replicate exactly elsewhere); cost ceiling (hyperscaler economics on sustained workloads); sovereignty implications.

Private Cloud Platform fits when the cost or sovereignty trade-offs make self-managed worth the operational investment. For early-stage companies without sovereignty pressure, hyperscaler-managed is still the right call.

When developer self-service is the right answer

Strong fit:

• 200+ engineers across 5+ product teams
• Time-to-environment is currently 2-6 weeks; target is hours
• Existing platform team is overwhelmed by ticket volume rather than building paths
• Multi-tenant isolation matters (regulated data, BU separation, or service-provider model)
• Investment in platform engineering is supported by engineering leadership

Marginal fit:

• 100-200 engineers with growing platform pain but limited budget; start with Cozystack Enterprise Support and scale into the Ænix Private Cloud Platform as the team grows
• Strong existing in-house platform with specific gaps — partial engagement may fit better than a full Private Cloud Platform build

Poor fit:

• Under 50 engineers, single product team; DevOps-only is right
• Hyperscaler-managed Kubernetes meets all requirements; sovereignty not a concern

Engagement structure

• Discovery call (30 min, free) — fit assessment
• Platform Readiness Assessment (14- or 28-day, IDP workstream emphasised) — current-state time-to-environment, maturity score, recommended golden paths
• Pilot deployment (3-6 months) — Cozystack platform + 3-5 golden paths + 2-3 pilot product teams onboarded
• Full build (within the 3-12 month Private Cloud Platform build, depending on scope) — platform expanded to the full engineering organisation, all targeted golden paths shipped
• Support subscription (ongoing) — Plus or Enterprise support tier for 24×7 escalation (see /pricing/)

Engagement size: project plus support subscription, quoted per RFP.

Where to dig deeper

• Private Cloud Platform landing — feature list, product-specific FAQ
• Internal Developer Platform services — engagement details
• Platform Engineering services — broader scope
• Developer self-service solutions — buyer-trigger landing
• Internal developer platform examples — 6 patterns — the six production IDP patterns
• Internal developer portal vs platform — Backstage’s place in 2026

You ship a managed Postgres or ClickHouse cluster to a tenant in two minutes. Now you need to charge them — by the minute, broken down by CPU, memory and disk, reconcilable to the cent. AWS RDS does this. GCP does this. On your own metal? Until now: a custom Prometheus script, a monthly spreadsheet export, and a sales call to explain “why is the number this big”. Ænix Billing closes that gap.

What Ænix Billing is

A native Kubernetes extension API server that exposes the billing.aenix.io/v1alpha1 API. You query it the same way you query the regular Kubernetes API — with kubectl, kubeconfig RBAC, your existing tooling — and it returns a structured UsageReport for any tenant, any workload, any time window.

Under the hood:

• Cozystack Workload CRD — every managed service (Postgres primary, Redis sentinel, ClickHouse shard, KubeVirt VM, Kubernetes worker, S3 bucket) is one Workload object.
• Billing controller — turns Workload state into Prometheus metrics: operational lifetime, owner tenant, kind/type, resource reservations.
• VictoriaMetrics — stores metrics with stream aggregation. (Ænix and Cozystack standardise on VictoriaMetrics + VictoriaLogs, not Prometheus/Loki.)
• Billing API server — serves the API, computes the definite integral of reservations over the requested time window.

The shape: a thin extension-API server, a small controller, your existing metrics store. No new dependency to operate. Your platform team already runs all of these components.

What gets billed

Every consumer (a single Postgres pod, a Redis instance, a ClickHouse replica) is reported with:

• vCPUHours or CPUHours — switchable, depending on whether you bill physical or virtualised CPU
• MemoryGiBHours (or decimal MemoryGBHours)
• EphemeralStorageGiBHours
• PersistentVolumeGBHours — partitioned by storage class (NVMe vs HDD vs replicated remote)
• IPAddressHours — derived from MetalLB IP pool labels
• LifetimeHours — pure on-time, independent of reservations
• S3StorageGBHours and S3PhysicalStorageGBHours — for object storage tenants

Workload metadata travels with the report — kind: postgres, type: master vs replica, the owning tenant, custom Cozystack labels — so pricing rules can be as fine-grained as you want. Charge replicas at half the rate of primaries. Charge NVMe-backed volumes 3× HDD-backed. Apply a 20% discount to a strategic tenant. The data is there; the policy is yours.

How you query it

A UsageReport is a Kubernetes object. You build a request like any other CRD:

apiVersion: billing.aenix.io/v1alpha1
kind: UsageReport
query:
  tenant: tenant-acme
  includeSubTenants: true
  startTimestamp: 2026-04-01T00:00:00Z
  endTimestamp:   2026-05-01T00:00:00Z

kubectl create -f report.yaml -o yaml

You get back the same object with report.consumers[] populated — one entry per Postgres replica, Redis pod, ClickHouse shard, K8s worker — each with its own consumption breakdown.

• includeSubTenants: true recursively pulls every nested tenant
• excludeTenants: filters internal namespaces out
• workload: postgres-myapp-1 scopes to a single cluster
• Time-window the report to a billing month, a single day, or a tenant’s lifetime — the API accepts any RFC3339 interval

The same kubectl get mechanism that pulls a list of pods now pulls a tenant’s invoice line items. RBAC works the same way — the billing team gets a kubeconfig scoped to the billing API; nothing more.

Why this matters for hosting providers

You already host managed Postgres, Redis, Kafka, ClickHouse, S3 and Kubernetes via Cozystack. With Ænix Billing you get the missing piece — accurate per-tenant, per-workload, per-resource accounting that plugs into your invoicing. No spreadsheets. No quarterly reconciliation. No “why is my bill big” arguments with customers.

Three things change for the operations team:

1. Billing is no longer a separate system. Same API as the rest of the platform. Same RBAC. Same kubectl. Nothing new to learn or run.
2. Disputes drop. The line items in an invoice map one-to-one to the resources Cozystack actually scheduled. The customer can run the same query, see the same numbers, reconcile to the cent.
3. Pricing experiments are cheap. Want to A/B test charging by MemoryGiBHours instead of vCPUHours for a specific service tier? Don’t change billing pipeline code — change the policy that consumes the same report.

How it differs from AWS RDS / GCP billing

AWS RDS and Cloud SQL bill per-minute against a managed-service price list — the provider owns the price list and the customer accepts the line items as authoritative. You are that provider on Cozystack. Ænix Billing gives you the same per-minute granularity, with the pricing model and the resource taxonomy under your control. The data is exposed, the policy is yours.

For hosting providers running on bare metal — Hetzner, OVH, regional data centres, sovereign cloud builders — this is the difference between competing with hyperscalers on capability vs. competing on price. The capability has been the gap.

Distribution

Ænix Billing is a proprietary Ænix module, delivered alongside Cozystack to customers with an Ænix subscription (for example Ænix Public Cloud Platform). The Cozystack platform stays Apache-2.0 and CNCF-governed; the billing layer is not part of it.

If you run a hosting business or a private cloud on Cozystack and want billing working out of the box, book a discovery call — we’ll talk scope, pricing, and rollout.

Join the community

• Cozystack — cozystack.io
• GitHub — github.com/cozystack
• Telegram — t.me/cozystack
• Kubernetes Slack — #cozystack (need an invite? slack.kubernetes.io)
• Community meetings — calendar