GPU Memory Calculator
Before launching a training run on a large language model, it helps to know whether the model, its optimizer and its activations will fit on the GPUs at hand, and which parallelism settings make them fit. This calculator answers that question for training and for inference, and shows the formula behind each number.
Main results
- Breaks training memory down into parameters, gradients, optimizer states and activations, with CPU and NVMe offloading, activation checkpointing and multi-node setups.
- Covers inference with the KV cache, its quantization options (INT8, FP8 and INT4), tensor parallelism and a throughput estimate.
- Ships preset configurations for models such as LLaMA 2, GPT-3 and Mixtral, and exports configs for Accelerate, Lightning and Axolotl.
- Runs as a web interface with formula explanations, as a command-line tool and as a Python API.