8xH100 inference on your own bare metal

A fast-growing developer of a mass-market mobile photo/video editing app moved its AI inference off a rented per-hour GPU cloud onto its own bare-metal 8xH100 server — keeping the familiar cloud UX (API, queues, autoscaling, model storage, monitoring) but on hardware it owns and controls. On Cozystack, running on generic Linux/k3s, the platform reached production traffic in about two months and delivers 2-3x better GPU efficiency than the previous rental.

About the project

The client is a fast-growing developer of a mass-market mobile app for creative photo and video editing. Several of its headline features — background removal and replacement, beautification, visual effects — are powered by the company’s own AI models rather than third-party APIs.

Those models ran on a rented public GPU cloud, billed by the hour. As usage grew the rental became expensive and capacity-capped: cost scaled linearly with load, and peak throughput was bounded by what the provider would hand out. The client decided to move inference onto its own bare-metal GPU server, while keeping the “cloud” developer experience its team was used to — an API, task queues, autoscaling, model storage and monitoring.

Goals and objectives

• Repatriate AI inference from a per-hour rented GPU cloud onto an owned bare-metal 8xH100 server, cutting cost and lifting the capacity ceiling.
• Preserve the cloud-native UX: API gateway, task queues, autoscaling, model-weight storage, full monitoring.
• Split responsibilities cleanly: the client writes and owns the ML business logic; Ænix owns architecture, DevOps and 24x7 support of the GPU infrastructure, Kubernetes and monitoring.
• Keep model weights and user data on infrastructure the client controls (data locality).
• No technology lock-in — an explicit, non-negotiable requirement: an all-open-source, CNCF-aligned stack with no proprietary control plane.

Proposed solution

A single 8xH100 server, layered top to bottom, turns owned bare metal into a private GPU cloud with an isolated tenant boundary:

• Client ML workers — the customer’s inference models, queue consumers and synchronous/asynchronous handlers, running as pods inside the tenant.
• Nested Kubernetes (“main”) — the GPUs are passed through to the tenant VM; the NVIDIA GPU Operator runs inside the VM for driver and device management, with ingress in front.
• Isolated tenant — a dedicated etcd, secrets, container registry and monitoring, so the workload never shares a control plane with anyone else.
• Cozystack on k3s / generic Linux — LINSTOR for storage, Cilium + KubeOVN for networking, KubeVirt for virtualization, GPU passthrough via vfio-pci, MetalLB for service publishing.
• Bare metal — 8x NVIDIA H100 80GB with NVLink and 2TB RAM.

Inference runs as two complementary pipelines. Asynchronous: API gateway → RabbitMQ queue → GPU ML workers → webhook callback. Synchronous: HTTP inference endpoints autoscaled with KEDA on nginx-ingress request rate (RPS), using VictoriaMetrics as the metrics source.

Execution: new requirements and how we handled them

• Talos to k3s, without losing manageability. The provider handed over SSH-only access — no console, no IPMI — which makes installing immutable Talos Linux impossible. We built a generic-Linux Cozystack on k3s over Ubuntu instead. The full platform (LINSTOR, Cilium/KubeOVN, KubeVirt, GPU passthrough, monitoring) works identically; nothing was given up.
• GPU passthrough of all eight H100s. Every H100 is handed to the KubeVirt tenant VM via vfio-pci. The classic “nvidia driver vs vfio-pci” race at boot — where the host driver claims a card before vfio can — was solved with an initramfs driver_override, so the devices land in the VM deterministically.
• RWX storage for shared model weights. Many worker pods need the same model weights concurrently. We provided shared read-write-many storage via a CSI wrapper plus NFS-Ganesha; the fix was contributed upstream to Cozystack.
• Traffic-based autoscaling. Synchronous inference workers scale with live demand using KEDA driven by nginx-ingress RPS metrics from VictoriaMetrics. The metrics-path fix that made this reliable also went upstream.
• GPU density. To pack more inference onto each card, we enabled GPU sharing via HAMi / HAMi fractional sharing, so several jobs can share one physical H100.

Results and current state

• In about two months the platform went live: end-to-end QA passed and all production inference traffic moved onto the client’s own cluster, where it has run stably since.
• Both pipelines are in production — asynchronous RabbitMQ jobs with webhook callbacks, and synchronous HTTP inference autoscaled on traffic.
• Grafana and VictoriaMetrics provide monitoring, with centralized logging across the platform.
• The customer runs its ML business logic on infrastructure it owns, at 2-3x the GPU efficiency of the previous per-hour rental, with no proprietary lock-in.

What’s next

• Grow the GPU-server fleet beyond the first node.
• GPU partitioning (HAMi / MIG) for higher inference density per card.
• A dedicated Harbor registry for heavy (~100GB) model images.
• Expand self-service on top of the multi-tenancy model.

Why this case matters

This case study is published in anonymized form (Tier-3 evidence): the customer is described by profile, not by name. A customer reference is available under NDA on request — talk to Ænix sales.

Ænix is the team behind Cozystack — a CNCF project (Sandbox today; Incubating expected late summer 2026), Apache 2.0. Ænix commercializes it as Ænix Platform, as three platforms on one engine — Public Cloud, Private Cloud and AI — that combine rather than exclude each other.