How Many GPUs Do You Need for LLM Training? The Complete Calculator
Model Size to GPU Requirements
The relationship between model parameters and GPU memory is straightforward at a high level but nuanced in practice. A model with N parameters in FP16 requires approximately 2N bytes just for the weights, plus 2-4x additional memory for optimizer states (Adam), gradients, and activation memory during training. For FP8 training, weight memory is halved but optimizer states typically remain in higher precision, reducing the total memory requirement by roughly 30-40%.
For a 7B parameter model in BF16, you need approximately 28-56GB of GPU memory depending on batch size and sequence length, making a single H100 or MI300X sufficient. A 70B parameter model requires 280-560GB, meaning you need 4x H100s (320GB total) with tensor parallelism or 2x MI300X (384GB total) for more efficient single-node training. A 405B parameter model requires 1.6-3.2TB, demanding 8-16x H100s or 8-12x MI300X accelerators depending on your parallelism strategy.
- 7B parameters: 1x H100 or 1x MI300X (single GPU training feasible)
- 13B parameters: 2x H100 or 1x MI300X (MI300X fits in single GPU)
- 70B parameters: 4x H100 or 2x MI300X (tensor parallelism required for H100)
- 130B parameters: 8x H100 or 4x MI300X (pipeline parallelism recommended)
- 405B parameters: 16-32x H100 or 8-12x MI300X (multi-node training required)
Parallelism Strategies: Data, Tensor, and Pipeline
Data parallelism replicates the full model on each GPU and splits the training batch across GPUs. It scales almost linearly but requires each GPU to hold the complete model in memory — limiting it to models that fit on a single accelerator. Tensor parallelism splits individual layers across GPUs, reducing per-GPU memory but requiring high-bandwidth communication for every matrix operation. NVLink's 900 GB/s on H100 or Infinity Fabric's 896 GB/s on MI300X makes tensor parallelism efficient within a single node.
Pipeline parallelism splits the model into stages across GPUs, with activations flowing forward and gradients flowing backward through the pipeline. It requires less communication bandwidth than tensor parallelism but introduces pipeline bubbles that reduce GPU utilization to 80-90% depending on micro-batch configuration. The optimal parallelism strategy for most production training runs combines all three approaches: tensor parallelism within nodes, pipeline parallelism across nodes, and data parallelism for overall throughput scaling.
- Data parallelism: best for models that fit on single GPU, near-linear scaling
- Tensor parallelism: split layers across GPUs, requires high-bandwidth interconnect
- Pipeline parallelism: split model into stages, 80-90% GPU utilization
- Most production runs combine all three: TP + PP + DP in a 3D parallelism strategy
- NVLink (900 GB/s) and Infinity Fabric (896 GB/s) are required for efficient tensor parallelism
Cost Estimation and Time-to-Train Analysis
Assuming cloud pricing of approximately $3.50/hour for H100 SXM and $4.00/hour for MI300X on major cloud providers, training a 70B parameter model on 8x H100s for 30 days costs approximately $20,160 in compute alone. The same training run on 4x MI300X at slightly lower per-GPU throughput but fewer GPUs required comes in at approximately $11,520 — a 43% cost reduction. For a 405B parameter model, expect $150K-$300K in compute costs for a single training run lasting 2-4 weeks on 32x H100s.
On-premise ownership dramatically changes these economics. At a purchase price of $30,000 per H100, the hardware breaks even with cloud pricing at approximately 8,570 GPU-hours — roughly 120 days of continuous training. For organizations running training workloads consistently, on-premise deployment typically yields 40-60% lower cost-per-training-hour over a 3-year depreciation cycle when factoring in power, cooling, and staffing overhead of approximately $0.50-0.75/GPU/hour.
- Cloud H100: ~$3.50/hour, 70B training on 8x H100 for 30 days = ~$20K
- Cloud MI300X: ~$4.00/hour, 70B training on 4x MI300X for 30 days = ~$11.5K
- 405B parameter training: $150K-$300K on 32x H100 for 2-4 weeks
- On-premise H100 breaks even vs cloud at ~120 days continuous use
- On-premise total overhead: $0.50-0.75/GPU/hour including power, cooling, staffing
Key Takeaway
Contact our engineering team for free technical consultation and workload-specific benchmarking across all accelerator platforms.
Related Products
Browse chips and systems mentioned in this article
Related Articles
Continue exploring our technical library
AI Chip Landscape 2026: NVIDIA, AMD, Intel Compared
A comprehensive analysis of the current AI chip market. We compare NVIDIA H100/H200/B200, AMD Instinct MI300X, and Intel Gaudi 3 across performance, memory, and ecosystem.
AMD Instinct MI300X Deep Dive: Architecture and Performance
An in-depth look at AMD's flagship AI accelerator, the MI300X with CDNA 3 architecture, 192GB HBM3 memory, and ROCm software ecosystem.
GPU Buying Guide 2026: How to Choose the Right AI Accelerator for Your Workload
A complete decision framework for selecting the right GPU in 2026. Covers training vs inference needs, memory vs compute tradeoffs, budget tiers, and vendor comparisons.