An NVIDIA A100 on AWS costs roughly $3.50/hour on-demand. Run 8 of them 24/7 for training workloads, and you're looking at $20K+/month in GPU compute alone. At that scale, on-premise bare metal starts looking very attractive — if you can manage the infrastructure.
We've set up GPU clusters in data centers for several AI-focused companies. Here's what the architecture looks like, what the pitfalls are, and when it actually makes financial sense to go bare metal.
When bare metal GPUs make sense
The break-even point depends on your utilization. If your GPU utilization is consistently above 60%, bare metal is almost certainly cheaper. Below 40%, cloud spot instances are hard to beat. The gray zone (40–60%) depends on your tolerance for spot interruptions and your workload's ability to checkpoint.
The trap is that almost nobody knows their real number before they buy. The figure people quote in the business case is the utilization they expect once the team has grown and the pipeline is mature — not the one their current jobs would produce. If you are weighing this up, the cheapest diagnostic available is to run DCGM exporter against your existing cloud GPUs for a month and look at the actual SM occupancy rather than instance hours. Instance hours tell you the GPU was allocated. Occupancy tells you it was working, and the gap between the two is usually where the business case quietly fails.
Two patterns we see repeatedly on that data. Research teams book GPUs interactively and leave notebooks attached overnight, so allocation looks close to total while real occupancy is a fraction of it. And inference fleets sized for peak sit mostly idle between peaks, which is the case cloud autoscaling genuinely handles better than a rack you have already paid for.
- Training workloads that run for hours or days → bare metal wins
- Inference with consistent traffic → bare metal wins
- Burst inference or experimentation → cloud wins
- Hybrid — steady-state on bare metal, burst to cloud → optimal for most
The architecture
Hardware layer
A typical setup we deploy:
- 4–8 GPU nodes (NVIDIA A100 or H100, 4–8 GPUs per node)
- High-bandwidth networking (InfiniBand or 100GbE RoCE for multi-node training)
- Shared storage (NVMe-oF or parallel filesystem like BeeGFS/Lustre for datasets)
- Management nodes (2–3 nodes for Kubernetes control plane, monitoring, storage controllers)
Kubernetes with GPU support
We use vanilla Kubernetes (kubeadm or RKE2) with:
- NVIDIA GPU Operator — automatically installs and manages GPU drivers, container runtime, and device plugins across all nodes
- NVIDIA Network Operator — manages RDMA and GPUDirect for multi-node training
- Time-slicing or MIG — for inference workloads, partition A100s into smaller virtual GPUs
- Topology-aware scheduling — ensures multi-GPU jobs land on GPUs connected via NVLink rather than PCIe
ML platform layer
On top of Kubernetes, we deploy:
- Kubeflow or MLflow for experiment tracking and pipeline orchestration
- JupyterHub for interactive notebooks with GPU access
- Volcano or Kueue for batch job scheduling and fair-share queuing
- Prometheus + DCGM Exporter for GPU utilization, temperature, and memory monitoring
- Harbor for container image registry (large ML images shouldn't traverse the WAN)
Common pitfalls
1. Underestimating networking
Multi-node training (e.g., training a large model across 32 GPUs on 4 nodes) is bottlenecked by inter-node communication. Standard 10GbE Ethernet won't cut it. You need InfiniBand or 100GbE with RDMA. This is often the most expensive and hardest-to-get-right part of the setup.
It is also the decision you are least able to revisit. Compute can be added a node at a time; the fabric is chosen once, cabled once, and changing it later means touching every node in the cluster. The practical choice is between InfiniBand, which is the better-trodden path for multi-node training and brings its own subnet manager and tooling, and RoCEv2 over Ethernet, which lets you keep one network team and one set of switches but puts the burden on lossless Ethernet configuration — priority flow control and ECN tuned correctly on every hop.
RoCE's failure mode is the one to understand before choosing it: when the lossless configuration is subtly wrong, nothing errors. Traffic falls back to retransmission, collective operations slow down, and the symptom reaching you is "training is slower than it was last week" with no alert attached. Budget time for NCCL bandwidth tests across every node pair at build time, keep the results, and re-run them after any switch firmware change — otherwise you have no baseline to compare against when somebody reports that exact complaint six months later.
2. Storage performance
ML training reads datasets in tight loops. A slow storage layer means GPUs sit idle waiting for data. We typically deploy a parallel filesystem (BeeGFS or Lustre) on NVMe drives with at least 10 GB/s aggregate read throughput per training node.
3. Power and cooling
An 8xA100 node draws 5–6 kW. A rack of 4 such nodes draws 20–24 kW. Most standard data center racks are provisioned for 6–10 kW. You need high-density racks with adequate cooling — often liquid cooling for the GPU nodes. Plan this with your data center provider before ordering hardware.
4. Drivers and firmware are the actual operational load
The hardware arrives and the interesting problems are over; what remains is a version matrix that has to stay consistent across every node. The GPU driver, the CUDA runtime in each container image, the kernel, the fabric drivers and the node firmware all have supported combinations, and the set that works is narrower than the set that boots. The NVIDIA GPU Operator removes most of the per-node toil here — it builds and loads the driver, manages the container toolkit and the device plugin, and keeps new nodes consistent with existing ones — but it does not decide when you upgrade, and it does not drain a node holding a seven-day training job.
Plan upgrades as a scheduled, drained, one-node-at-a-time operation with the job queue paused, the same way you would treat a database version bump. The failure people actually hit is more mundane than a driver bug: a node reboots after a firmware update, comes back with a slightly different driver than its neighbours, and jobs scheduled across the two fail in ways that look like application errors. Pin the versions, make the matrix explicit in the repository, and treat a node that has drifted from it as failed rather than degraded.
5. Sharing one cluster between teams
A single team with one workload needs very little scheduling. The moment a second team shares the cluster, the questions change from technical to allocative: who gets the eight-GPU node, what happens to a queued job when a higher-priority one arrives, and whether an idle reservation can be reclaimed. Kubernetes alone does not answer these well, which is why batch schedulers — Volcano or Kueue — appear in the stack above. They provide gang scheduling, so a multi-GPU job either starts with all of its GPUs or waits rather than deadlocking half-placed, and fair-share queues so one team's backlog does not starve another's.
Partitioning is the other half. MIG carves an A100 or H100 into hardware-isolated instances with their own memory and compute slices, which suits inference and interactive notebooks where predictability matters more than peak throughput. Time-slicing shares a whole GPU between processes without isolation, which is cheaper to set up and fine for development, but one job's memory allocation can starve another on the same card. Choosing time-slicing for a shared research cluster and then diagnosing intermittent out-of-memory failures is an avoidable fortnight.
None of this is a technical problem for long. It becomes a policy one, and the cluster works far better when someone has written down what the priorities are before the first contended week rather than during it.
Cost comparison
For a typical 8xA100 setup running 24/7:
- Cloud (AWS p4d.24xlarge on-demand): ~$24K/month
- Cloud (reserved 1-year): ~$16K/month
- Bare metal (colocation + hardware amortized over 3 years): ~$6K–$8K/month
The bare metal option is 2–3x cheaper at high utilization. But it requires upfront capital ($200K–$400K for hardware) and ongoing management.
That last figure is also the one most often quoted without its contents. A colocation bill plus amortized hardware is not the running cost of a cluster: add spare parts held on site, because a failed GPU on a three-day RMA is a node out of the pool; add the fabric and storage layers, which are not optional on a training cluster; and add the person who is on call when a node drops out of a distributed job at 3am. None of these change the conclusion at genuinely high utilization — the gap is wide enough to absorb them. They do change it in the gray zone, which is exactly where most teams are when they first run the numbers.
That ongoing management is where we come in — we handle the infrastructure so your ML team can focus on models.