When dedicated GPU pays back
Hyperscaler GPU is competitive for sporadic workloads. For sustained workloads, the math shifts. Crossover point varies by GPU class:
- H100 sustained: crossover ~30% utilization
- L40S sustained: crossover ~40% utilization
- A100 second-hand market: crossover ~25% utilization
- Blackwell: depends on availability and pricing
For inference serving 24/7 at sustained traffic, owning the GPUs is usually 30-60% cheaper after Year 2.
What goes in an AI platform
- GPU compute — sized by workload, expandable
- Model serving stack — vLLM, Triton, or alternatives
- Inference orchestration — request routing, batching, autoscaling
- Vector store for RAG (where applicable)
- Multi-tenant for customer-facing AI products
- Observability for GPU utilization, latency, costs
- Operations for 24×7
Common architectural decisions
Decision 1: VM-based vs container-based GPU
KubeVirt with VFIO passthrough or NVIDIA vGPU for VMs (data-science teams that need full VMs); the NVIDIA GPU Operator plus HAMi fractional sharing for containers (production inference). Cozystack supports both.
Decision 2: Self-hosted vs cloud GPU
Self-hosted for sustained; cloud for sporadic. Hybrid for elastic.
Decision 3: Open-weight vs proprietary models
Open-weight (Llama, Mistral, Qwen, DeepSeek) for sovereignty + cost; proprietary for specific capability needs. Most production deployments are open-weight in 2026.
Decision 4: Inference-only vs fine-tuning vs training
Most AI startups: inference + occasional fine-tuning. Training at scale is rare in startup context.
Test yourself: AI platform for startups
5 questions · ~2 min